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In the spring o£ 1969 three organizations, The Council of the 
Great City Schools; TesLchevs College, Colmbia tJniveristy; and the 
School District of Philadelplu a, were brought together to di^scuss the 
possibility of a research training program i^ich could upgrade re- 
search and evaluation activities in the nation's largest school 
districts. A joint proposal was prepared and the project f^mded, 
with the Council serving as priine contractor, and the other two 
organizations inrolved in a coopeiative arraiigCTient. 

Ihe general purpose was to field test a set of activities which 
could upgrade the quality and effectivenc^-ss of research and evaluation 
activities in large urban school districts. ITiree aspects of this 
problem wera addressjed: 1) The need for better conceptml irodels for 
evaluation in the school setting^ 2) Hie need for specially -trained 
entry-level persomiei for school research division, and 3) The need 
for expanding; the knowledge and skills of existing staff. 

Problem one, the need for better conceptual models, was handled 
by Teachers College. Under the general direction of Francis A. J. 
lanni. Teachers College staff developed what might be described as an 
anthropological or "naturalistic" appwach to evalua;tion. The model 
is designed for application in real-life settings and does not depend 
on eDq)erimental controls. It relies heavily on observation as an 
information source; its data are largely qualitative in nature, i.e,, 
they are produced through jud.gments; it requires a team whose members 
have different types of expertise and different levels of eDq)erience; 
and it is oriented toward the support of administration and of decision 
making in the operating school district. 

In addition to developing and field testing the model, Teachers 
College designed a program to train people in its use. Two broad 
courses were worked out in detail, and descriptions of them are in- 
cluded in the report which ac^companies this summary. Briefly, the 

courses are: 

One, An Overview of the Theoiy of Evaluation and the Rationale 
and Design of Evaluative Studies. 

TVo, Qualitative Date--its Collection and Analysis: An* 
Approach to Problem Solving. 

Problem two, the need for specially trained entry- level personnel 
for school research divisions, was addressed The School District of > 
Philadelphia through its Office ^of Research and Evaluation. The 
approach was based on the premise that the school research specialist 
needs, in addition to a wide-ranging set of technical .skills, ah under- 
standing of the context in which he will work and of the ways skills 
can be used effectively in the particular feal-life setting. 'Ihe 
prograin had the following components: 
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An on-the-job trainxrig" eiiperience in which the trainee 
served one semester in each of the three divisions 
(Instructional Research and Development, Administra- 
tive and Sixrvey Research, and Testing) o£ the District's 
Office of Research and Evalmtion. 

An Academic program in vMch trainees piirsued a Master 
o£ Education degree in Educational Research and Measure- 
ment at the University o£ Pennsylvania. 

A summer vrorkshop in which trainees pursued individual 
projects using simulated data, and in which they were 
trained in computer applications. 

A seminar in which trainees had the opportunity to 

share experiences and concerns and to investigate pertinent 

topics o£ interest* 

'An independent project jji which the trainee had the op- 
portunity to carry out a coirq^lete study under the 
guidance of a knowledgeable sponsor in the School 
District. 

The on-the-job and university conpDnents were carried out 
simultaneously, providing for an interaction of formal skill de- 
velopment and real -life experience. 

Seven trainees participated in the Philadelphia portion of 
the project, ihree took jobs in city research division staff, was 
handled by the Council of the Great City Schools. (The Council 
as prime contractor, also had responsibility for overall coor- 
dination of the program.) 

A highly individualized approach, was used by the Council. 
Persons with some, background^ and experience in research were re- 
cruited and brought' to the Council offices in Washington, D.C. • 
Interns came from the Cleveland Public Schools, the School District 
of Philadelphia, the University of Missouri in St. Louis, the 
University of Wisconsin, and the University of South Florida Each 
intern selected one or more areas of special interest and he 
worked within the.Cotmcil and its 21 meniber districts to help con- 
ceptualize and implement in^jrovement strategies . Siibstantive areas 
selected were: large city testing problems, performance contracting 
and accountability, career development, special evaluation pro- 
blems encountered in Title I of the Elementary and Secondary 
Education Act, education of young children, and exchange of e^^lua- 
tive information among large urban school districts. 

Through the Council ei^erience, the interns had a unique 
opportunity to work With the most knowledgeable persons .within the 
nation's largest school districts, with representatives of the U.S. 
Office of Ecucation and other leading national experts in the 
respective areas of concern. Ihey also attended meetings of the 
Great City superintendents in which a broad range of issues confronting 
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the large urban school districts were discussed. 



Five interns v?ent through the Council portion of the program. 
Three of these took middle-level positions in research divisions 
in Council meinber districts at the program' s conclusion. One took 
a position with a priA^te educational consulting firm, and the 
other went to work for a state agency. The ijnportant point is that 
each emerged with a higher-level and more responsible position 
than he had previsouly held, and each remains in the field of edu- 
cation. 

Overall, the Great Cities Research Training Project can 
be counted a success . It generated and successfully tested jicdels 
for solving some of the most difficult school evaluation problems. 
Within this larger context, it identified a number of procedural 
problems and suggested ways these should be overcome. Finally, it 
provided a success experience for about 80 percent of the persons 
who entered it either as trainees or interns. It did not solve the 
evaluation problems of the large urban districts, of course, but it 
took an important first step toward developing solutions. 
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I. DESCRIPTION OF THE PROJEa 



A. Background 

The Need . Research and evaluation has taken on greatly 
iticreased importance in large city school systems in recent years. 
From the local perspective, school systems have become increasingly 
aware of the need to improve their management and decision-making 
capabilities; in fact, they find themselves in a crisis of such 
proportions that they must improve or face their demise as viable 
institutions within the society. At state and national levels, there 
have been requirements for systematic evaluation which shows actual 
program results and \\*Lich serves as the basis for allocation of resources, 
and evaluation demands are becoming more stringent. Local, state, and 
national needs for valid evaluative infoimation cannot be met, in 
large city school systems at least, without significantly strengthening 
the capabilities of school research and eva.luation divisions. 

The nature of the need for improved research and evaluation 
capability in large school systems has become increasingly apparent 
since 1965. Experience, beginning primarily with implementation of 
Titles I and III of the Elementary and Secondary Education Act of 
1965, demonstrated decisively that existing research organizations 
were not capable of meeting the demands placed on them, and methods 
of increasing the capability became a prime concern of school research 
directors, as well as superintendents and other high=level administra- 
tors. University researchers also recognized the need; they had ex- 
perienced great frustration in attempting to assist school districts 
in evaluation efforts, and many turned their attention to the develop- 
ment of new methodologies. At the most general level, the problem 
was an inability to produce infoimation which was useful to decision 
makers and which could help in improving program. 

As recently as six years ago, the major responsibilities of 
educational research divisions in public school systems were limited 
largely to running testing programs and to administrative data gathering. 
As^ the situation changed and demands increased, public school research 
organizations became more and more aware that they desperately needed: 
(1) more workable models for the kinds of evaluation activities demanded 
of them, (2) more adequately trained personnel to design and conduct 
evaluative activities. 

Planning of the Project Against this background, three 
organizations in the Spring of 1969 submitted separate proposals 
addressed to particular aspects of the problem, for funding under the 
Research Training Program of the U. S. Office of Education (OE) . The 
three Were the Council of the Great City Schools, Teachers College 
of Columbia University, and the School District of Philadelphia. OE 
officials recognized overlap in both purpose and methodology among 
the proposals and called representatives of each organization together 
to discuss a common effort. Joint planning ensued, and a single pro- 
posal was prepared and submitted and was subsequently funded. 



Because of its position as a consortium of the largest urban 
school districts, the Council o£ the Great City Schools was chosen as 
prime contractor, with a portion of project activities to occur 
within Council offices. The Office of Research and Evaluation of the 
School District of Philadelphia was to handle certain training activities 
under a subcontract, and Teachers College, Columbia, was to handle 
Ttiodel building and planning activities under a purchase of services 
agreement. The project was originally scheduled to run from August 
15, 1969 through August 15, 1970. An extension to June 30, 1971, was 
requested and granted. 

The Council of the Great City Schools was the natural agency to 
handle the prime contract because of its critical position as a source 
of information and as a facilitator of research activities in the nation's 
largest school districts. Furthermore, the Council is in a central 
position in regard to efforts to improve education for cultrually 
disadvantaged and econoirically deprived children. In addition to 
securing services from the Philadelphia schools and Teachers College, 
the Council was responsible for obtaining other consultant and uni- 
versity services required by the program. 

B. Objectives 

The project had these general goals: 

To study research and evaluation problems of special 
concern to large city school systems. 

To facilitate and coordinate activities designed to 
provide immediate solutions to some of these problems. 

To develop an ongoing internal researcli capability within 
the Council of the Great City Schools so that developing 
research and evaluation needs of large city school - 
systems can be met. 

To develop methods for the evaluation of qualitative 
variables in education and to develop training models 
for preparing researchers to use these methods. 

To develop and conduct an effective training program to 
meet the specialized research and evaluation needs of 
city school systems . 

To disseminate findings and practices which show promise 
of improving research and evaluation activities in the 
Great Cities. 

The Council portion of the project had the following specific 
objectives: 

lo survey the existing roles of researchers and evaluators 
in member cities and to determine the need for role 
. changes . 
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To assist the Philadelphia Schoo?.s and Teachers College 

in developing conceptual evaliiation models and to cooperate 

in field testing of these inodels. 

To conduct a training program for research interns which 
would develop competence to: 

Develop reliable instruments for gathering accurate 
information related to educational problems. 

Organize, analyze, and reduce raw information so 

that accurate and understandable reports can be prepared. 

Organize and conduct information gathering activit^ies 
through surveys, interviews, observations, pap'er-and- 
pencil instruments, and search of literature and 
other relevant documen 

Prepare proposals. 

Conduct basic statistical analyses and make tenable 
inferences from them. 

The Teachers College portion of the project had these objectives: 

To develop a conceptual model for the evaluation of 
qualitative variables at the public school level 
in education. 

To develop methods that will permit researchers to 
evaluate these variables in quantifiable terms and to 
rel ate' obtained results to other evaluation data. 

To develop a related training ntodel for the preparation 
of researchers in the use Df these methods. 

The Philadelphia portion of the project had the following 
objectives: 

To develop the design and analysis skills needed for 
the conduct of evaluation activities in public school 
districts. 

To develop knowledge of the political and organizational 
realities v^ich must be dealt with in conducting evalua- 
tion activities in school districts . 

To develop skills in analyzing and reporting evaluative 
information so that the decisions of relevemt school 
district staff are appropriately influenced. 

To develop the interpersonal skills necessary to suc- 
cessfully plan and conduct evaluation activities in 
school systems. 
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To develop, through on-the-job training, the ability to 
plan, implement, and conduct evaluation activities 
in school districts. 

To develop commitment to school district research and 
evaluation efforts, so that trainees remain in the public 
school setting after the program is completed. 

C. Responsibilities of Cooperating Agencies 

The Council of the Great Citv Schools --As noted, the Council 
had two major responsibilities in tne project. First, the Council was 
prime contractor and coordinated the total effort. This included 
monitoring the activities occurring in the Philadelphia schools and 
at Teachers College, Coluirbia University. The Council was the agency 
ultimately responsible, therefore, for meeting all project objectives. 

The Council con^onent was designed for interns with some pre- 
vious experience in the research and evaluation field. Interns, already 
with some capability for working in public schools, were to be given 
a broader perspective and to develop specific skill areas more highly. 
They would serve in the Council offices in Washington, work on pro- 
blems common to the largest school districts, help develop proposals, 
meet with U. S. Office of Education representatives, and have other 
experiences of this type. 

To conduct itw own component and to coordinate the other program 
segments, the Council was to employ a full-time Project Director and 
it was to be responsible for preparation of the final report of the 
project and for dissemination of project findings. 

Teadiers College Teachers College was responsible for 
generating both a theoretical framework, and a set of methods for evalua- 
ting qualitative variables in education with the same precision and 
rerlicability that characterize quantitative evaluation. Teachers 
College was further responsible for developing a training program for 
the preparation of research personnel in the use of its evaluation 
model. Pilot tef^ting and refinement of the model, and training of 
research interns in use oi he model were to be concurrent activities. 
TeaciiciS College was further to compile relevant literature on related 
research metliods and prepare a position paper on the general subject. 
Finally, Teachers College was to cooperate with the Council in tiie 
training of Council interns. 

The School District of Philadelphia The Office of Research 
and Evaluation of The School District of Philadelphia was responsible 
for conducting a specialized training program to prepare trainees for 
research and evaluation activities in city school systems. This training 
effort combined on-the-job experience in the Office of Research and 
Evaluation with academic training conducted by the University of Penn- 
sylvania. The Philadelphia portion of the program was designed to 
last some twenty-one months (two full academic years, plus a summer 
term), and trainees were to receive an M.S. degree in educational re- 
search, providing they met all requirements of the University. 



The Philadelphia coiUDonent was designed for trainees ivith no 
previous research and evaluation experience. The idea was to take 
promising young college graduates, with a year or two of teaching 
experience desirable but not required, introduce them to the public 
school research and evaluation field, and train them to operate ef- 
fectively in this setting. It was therefore of more lijnited scope 
than the Council component, where a national perspective was to be 
developed. 

Summary of Project Components --In summary, the three components 
of the project were all directed toward inq^roving the e\''aluation 
capabilities of large -city school districts but each component addressed 
a district portion of the overall problem. 

The Council of the Great City Schools took as interns 
persons viho already had training and experience in 
evaluation. It gave each intern the opportunity to 
pursue an area of individual interest in great depth 
and, in doing so, to work with knowledgeable personnel 
from the nation's largest school districts, the Office 
of Education, and other appropriate agencies. It also 
provided a national perspective on problems of urban 
education and on application of evaluation techniques to 
these problems. 

The School District of Philadelphia took trainees 
without previous experience in tlie evaluation field. 
Through a joint program of on-the-job training and 
formal course work. The trainees were prepared 
for regular staff positions in school research offices. 

Teachers College worked on development of a new 
evaluation model and on a program of training in use of 
the model . It focused on use of qualitative variables 
in evaluation design, an area which experience has 
shewn to be one of the most difficult in school district 
evaluation efforts. 



II. PROJECT ACTIVmES OF 
THE COUNCIL OF THE GREAT CITY SCHOOLS 



a. Starti:^) and Selection o£ Interns 



Project activities began on August 15, 1969, with Mr. Jack 
I. Marcussen o£ the Council staff serving as interim project director. 
Initial activities involved conpleting contractual arrangements with 
Teachers College and with the School District of Philadelphia, and 
beginning the search for a permanent project director. 

In searching for a director, the Council initially conpiled 
a list of some 65 persons across the nation. Further discussion 
and interviews led to the conclusion that it would be desirable to 
select a person with experience as a research director in one of 
the larger school districts, and the Coimcil finally focused its at- 
tention on the menibership of its own Research Steering Committee. This 
was a six- member group, elected by the research directors of the 
Council's member districts, which had played a primary role in develop- 
ment of the proposal. A Second round of intensive interviews was 
conducted, and the position was finally offered to Dr. John L. Hayman, 
Jr. , Executive Director of Research and Evaluation of the School District 
of Philadelphia. Dr. Ka/man accepted and assumed responsibility for 
the project on a full-time basis on January 15, 1970. 

After meeting with representatives of Teachers College and 
the Philadelphia schools to discuss progress to date and to outline 
next steps in the project, the director turned his attention to 
selecting interns for the Council's coTi5X)nent. 

The original plan had called for six interns, but increasing 
costs plus the need for an extension of the project caused by the 
delay in selecting a permanent director forced the number of interns 
to be reduced to five. Names of candidates were solicited from menfcer 
districts, as well as from the Office of Education ^ind from universities 
close to or in menier cities • A selection committee, consisting of 
the following individuals, was established: 

Dr. John L. Hayman, Jr. 
Project Director 

Mr. Jack I. Marcussen 

Council Staff and Former Interim Project Director 
Dr. Oscar HankLnson 

Principal and J?oxnjer Research. Staff Member 
The School District of iTiiladel^ia 

Interviews were conducted, and four interns were initially 
selected, as follows: 



Mr. Jerry Calendine 
Research Staff Memfaex* 
Cleveland Public Schools 

Mr. William Denton 

Former Principal and Doctoral Candidate in 

Educational Research 
The University of Wisconsin 

Mr. Jack Stenner 

Former Teacher and Master^ s Candidate 
The University of Missouri at St. Louis 

Mr. Edivard Whitney 
Research Staff Maiiber 
The School District of Philadelphia, and 
Doctoral Candidate in Educational Psychology- 
Wayne State Universit/ 

The initial four interns joined the Council staff in Februar)* 
and March of 1970. Interviews continued, and the fifth intern was 
selected in June. He was: 

Mr. James Van Or den 

Former Teacher and Doctoral Candidate in 

Educational Research 
The Uhiversity of South Florida 

As noted, an objective in the Council ^s conponent was to tailor 
each intern's experiences according to his needs and his particular 
research interests.. In line with this philosophy, each intern had a 
set of'imique experiences. Mr. Whitney chose to work in the area of 
information dissemination and utilization, Mr. Calendine was assigned 
to work with testing issues which confront large school systems, Mr. 
Denton worked with broad evaluation problems in large school systems 
and x^rith developing instructional systems in vocational and technical 
edcuation, Mr. Stenner worked with and became a leading authority in 
performance conti;acting and Mr. Van Orden was assigned to the problem 
area of packaging and presenting information in the most meaningful 
and usable way. 

In addition, a common set of experiences was provided all 
interns to meet the objectives of developing a broader national per- 
spective and developj-ng .knowledge of research and evaluation problems 
which confront all large city school districts . The interns met * 
regularly with Council staff, and they attended a special one-week 
session organized by Teachers College, in addition, they attended three 
research training seminars >Mch were conducted by the. Council under 
a different program, ''Improving Research Capabilities of Large City 
School Systems COE Grant No. OGE-0-9-203495-4432 1010])." The seminars ' 
will be explained in more detail later in this report. 



Experiences of Individual Interns 



As noted, each, intern had a set of unique experiences in the 
program, depending on his interests and his skill areas. A description 
of each intern's experiences follov^rs: 

1. Jeriy Calendine 

Mr» Calendine was a Research Assistant in the Division of Research 
and Development, Cleveland Public Schools, prior to his internship 
with the Council. Because of experience in the testing program in 
Cleveland and an interest in educational psychology, he was assigned 
to the area of testing. 

As early as it's Fall 1969 meeting, the Council's Board of 
Directors, composed of superintendents and Board of Education members, 
had identified testing as one of the areas of high priority to be 
investigated by the Council staff. Possible inadequacies of the test 
instruments for inner- city children; the inappropriateness of test 
norms for urb.an students ; and the relationships between testing and 
educational policies were expressed concerns. 

Mr. Calendine ^s internship at the Council was primarily focused 
upon these problems and issues surrounding standardized testing of 
children in public schools. The Washington location of the Council 
provided many nearby resources for stu<^ and input to this type of 
activity. The Office of Education, tlie Office for Civil Rights, the 
National Educational Association, and the Library of Congress placed 
at the Council's disposal excellent library facilities. Government 
officials, test publisher representatives, and public school testing/ 
research specialists from Council meniber cities cooperated in the 
study of testing issues. 

As a preliminary step in the investigation of the issues of 
standardized testing, Mr. Calendine, in cooperation with the project 
director and other Council staff, planned and conducted a survey of 
the 21 city school districts which then comprised the Council of the 
Great City Schools . The results of this survey (Appendix A) gave a 
better understanding of testing practices and helped focus major 
issues in the testing area. Also, it identified the publishers 
who provided the great majoidty of tests used in the largest school 
districts • 

With survey results in hand, Mr. Calendine and the Council staff 
arranged a series of meetings with representatives of the Office of 
Education^ the HEW Civil Rights Office, and with various persons 
from men4)er school districts. The focus was on a better understanding 
of needs and issues in testing. The Council 's Research Steering Com- - 
mittee met to provide guidance to the investigation. 

Follow-up meetings were held with test publishers in the attempt 
to develop a cooperative plan to deal with some of the issues of 
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standardized testing. One o£ the mos-^ significant outcomes was agree- 
ment by Science Research Associates to prepare special Great City norms 
for the tests it standardized in the spring of 1971. Other publishers 
agreed to cooperate in like manner in their won standardization or re- 
standardization efforts in later years. . To assist in the 1971 standar- 
dization, Mr* Calendine worked \n.tii SRA representatives to create an 
ethnic/racial matrix of schools within the Council membership. 

Mr. Calendine also assisted in the preparation of a proposal 
^vhich would provide resources for the Council to pursue its study of 
testing issues in greater depth. He participated in meetings with 
Office of Education and Civil Rights Office representatives in out- 
lining the proposal and in attempts to secure funding. 

As a result of his studies with the Council, Mr. Calendine has 
noted that testing problems result from misunderstanding and/or misdi- 
rected effort on the parts of at least three groups: School Administra- 
tors, Test Publishers, and Community members. 

^'The use of standardized tests/' he states, ''enables school 
administrators to objectively sample a measure of pupil performance 
with an instnmient constructed so as to provide: 

1. A common denominator from school building to school 
building . . . from one section of the country to 
another section; 

2. A test instrument with expertly developed and 
perfected items . 

"A serious problem has emerged in recent years for school ad- 
ministrators , and. it realtes to the selective population movement of 
middle class whites to the suburbs and the growing concentrations of 
large communities of minority groups in the central city. Using 
standardized tests that are constructed from a ^'representative national 
sample*^ of students, inner-city school administrators are now respon- 
sible to communities that in no way resemble the ethnic/racial com- 
position of the nation. At one time the concept of a national sample 
and a test geared to that national cultural norm may have had a legitimate 
defense, but the current policy is nearly untenable to minority groups 
that have been socially and culturally isolated.'* 

With regard to test publishers, Mr. Calendine notes a tendency 
to resist change with considerable 'Vehemence" and to place the blame 
elsewhere for the misuse of test results. "Despite numberous attempts 
to help them understand the issues being raised, test publishers con- 
tinue to insist upon drawing their 'representative national samples' 
o£ children ignoring the fact that their ^market' for these tests is 
the large cities . The argument that urban or local norms are provided 
does little to satisfy critics who point out tfie middleclass orienta- 
tion of standardized test items. Granted, test publishers miast accept 
or define some cultural standard on vhich to base their tests, but 
It is doiabtful by the very nature of the sample-selection processes 
whether test publishers have adequately researched test items to ensure 
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culture fairness/' 



The inajor problem so far as coiTiTnuni-ties are concerned, Mr. 
Calendine concludes, is their lack of understanding of vHiat tests are 
supposed to do and their use of test results in contradictory -l^^ays , • 
Coirmumity representatives will, on the one hand, note the cultural 
bias of tests and state with great conviction that obtained results 
are invalid. On the other hand, they will use these same results as 
a club, using them as an indication that the schools are i^ot doing 
their job well. One argument states that the tests are invalid; the 
other assumes that they are valid. 

At the end of his internship and year of study with the Counil, 
Mr, Calendine drew up the following list of recommendations with regard 
to testing: 

1, Superintendents c£ public school systems should 

ask test publishers for evidence that test items are 
being'tested and examined for cultural fairness, 

2» Public school testing personnel should perform their 
own item analyses of standardized test results, 
particularly where large concentrations of minority 
group children are found in their district, Sound 
judgments may then be made with documentation should 
the use of tests be examined in the courts, 

3, The purposes of standardized testing should be more 
clearly defined by school administrators. Do these 
tests in fact aid instruction? This should be weighed 
against teacher-made tests or locally- constructed 
tests ; where immediate feedback (discussion) may take 
place after the testing situation. In such instances, 
the poorest constructed test item may promote class 
discussion, clarification, and learning. After de- 
fining the role of standardized tests, a school 
system should be in a position to more reasonable de- 
cide the proper balance of types of tests to be ijsed. 

4. Implied in (3) above is a review of practices of 
testing. School administrators should be able to 
defend their practices with regard to the use and 
disclosure of standardized test results. This may, 
involve a review of student placement policies as 
well ?r5 procedures necessary to communicate with the 
press, civic, and educational organizations. 

Mr, Calendine returned to the Cleveland Public Schools as a 
Research Associate at the concliasion of his internship, 

2, William T. Denton 

Mr. Denton was in the doctoral program in educational research 
at the University of Wisconsin when he joined the Council as an intern. 
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His experience had included seven years as a classroom teacher and four 
years as a high school principal, l^ith the Council he took as special 
r interest areas general evaluation problems in lavge city schools and 
improvement of career education. 

When Mr, Denton arrived, the Cornell was in the midst of a 
study funded by the Office of Edcuation's Division of Compensatory 
Education and titled, 'Title I in the Great City Schools: An Analysis 
of Evaluation Practices and Exeirplary Projects." Because of his ex- 
tensive experience as a public school teacher and administrator, he 
was made a co -director of the project and had been included in earlier* 
planning sessions. The purposes of the project were ''to identify sue- ■. 
cessful Title I Projects whthin member districts (of the Council) 
determine some of the reasons these projects were successful, study 
Title I evaluation practices, and, from the perspective of local district 
personnel, identify special problems related to Title I evaluation and 
make recommendations for improving evaluation practices/'* 

In conjimction with this project, Mr. Denton first helped to 
organize and conduct a conference of Great City Schools representatives 
in Lake Placid, New York, in June of 1970. Mr. Denton was specifically 
responsible for collecting and summarizing information produced at 
conference sessions each day, for preparing an overall report of the 
conference, and for helping to integrate information produced by the 
conference with other infoimtion to produce the final report of the 
larger project. Other information had been collected through a series 
of site visits, interviews with research directors and project directors, 
and questionnaires completed by research directors and federal program 
directors . 

Mr. Denton worked with project staff and with the Council Research 
Director in analyzing information, drawing conclusions, and preparing 
a report which would meet the needs of the Division of Coir^jensatory 
Education. The report was coiT5)leted and delivered on time, and it has 
been cited as probably the best existing statement on Title I evaluation 
practices and problems. Mr. Denton contributed significantly to the 
success of the effort, and the insights be gained were invaluable in 
his own development as a p^-'jlic- school research and evaluation specialist. 

Mr. Denton's second major interest area with the Council was 
Career Education, The Council Board of Directors had made career edu- 
cation a priority concern and had directed the organization to renew 
its efforts in this area. (The last care3r education project had been 
completed in 1968.) Prior to Mr. Denton' i arrival^ meetings had been 
held with representatives of the Council, the Office of Education, and 
with the American Institutes of Research, which was conducting a develop- 
mental project in career education in Quincy, Massachusetts* It had 
been determined to hold a conference to define career education problems 
of the large urban school districts more precisely and to outline a plan 

^Hayman, John L., Jr., et.al, "Title I in the Great City Schools: An 
Analysis of Evaluation Practices and Exemplary Projects." Washington, 
D.C. The Couicil of the Great City Schools, 1970. (MIMEO.) p.l. 



,o£ action in career education for the Council. This conference was 
held about one inonth after Mr. Denton* s arrival, and he shared respon- 
sibility for -coll ectingaand analyzing the information prodticed. 

The primary outcome of the conference was a decision to seek 
funding for a large developmental project to .be conducted by the Council. 
The strategy was to identify the job families in greatest need of 
instructional iirprovement; determine which city would undertake res- 
ponsibility for each job family, derive sets of common (across city) 
objectives, devise a common strategy for instiTctional systems develop- 
ment, and, through a carefully coordinated and articulated effort, 
revamp career education in the urban school districts. The planning 
requirements for this kind of activity were, of course, enormous. 

A planning committee was established, with Benjamin Whitten, of 
the Baltimore City Schooks, as chiarman. Both the Executive Vice- 
President and the Research Director of the Council were closely in- 
volved. Mr. Denton served as a kind of secretary-research assistant- 
writer for the committee. He attended all committee meetings with 
Office of Education representatives, and he took notes and prepared 
minutes for meetings. He also did extensive library research on 
various aspects of the problem area. As efforts began to prepare 
drafts of a proposal, Mr. Denton was given various writing assignments. 

Reaction to early efforts ras favorable, and the Council was 
encouraged to prepare a proposal for a planning grant to facilitate 
the effort. Mr. Denton participated in discussions of the outline 
and content of the planning proposal, and he took major responsibility 
for writing of the proposal. The proposal was approved and a grant 
received. 

The funded career education planning project became operational 
after Mr. Denton had bee/i a Council intern for some eight months. 
Because of the capability he had demonstrated and of his willingness i to 
take responsibility, he was asked to serve as temporary co-director 
of the project. He held this position throu^ the remainder of his 
internship. 

Mr. Denton also assisted the Council in planning for and con- 
ducting a workshop on project planning and evaluation for the School 
District of Philadelphia, the workshop, held in December of 1970, 
was for principals and other middle -management personnel in one of the 
Philadelphia's ei^t administrative districts. Mr. Denton made a one- 
hour presentation at tlie workshop, and he helped to conduct the workshop 
evaluation^ 

After completing his internship, William T. Denton accepted 
a position as research associate with the Research Division of tlie 
Dallas Independent School District. He is currently serving with the 
Dallas schools. - 



3. Alfred Jackson Stenner 



Prior to his selection as a Council intern, MR. Stenner had 
been a teacher in a clinic for emotionally disturbed children in 
St. Louis and was enrolled in a master^s program at the University of 
Missouri at St. Louis, His undergraduate work had been in psychology 
and education, and his graduate work was in education administration. 
Mr. Stenner' s recoinmendation for the internship came through the St. 
Louis city schools, who had learned of him from representatives of 
the local branch of the state University. University faculty described 
him as "^he most outstanding student in educational administration in 
the history of their institution. 

His professional experience prior to joining the Council was as 
follows: 

1967- 68 Caseworker-Aid to Dependent Children Unit, 

Missouri Division cf Welfare. Primary res- 
ponsibility included certification for 
payment and case reviews. Area of respon- 
sibility, inner-city, St . Louis . 

1968- 69 Rous eparent- Evangelical CMldrens Home, 

Normandy, . Missouri . Primary responsibilities 
involved acting as surrugate parent for twelve 
girls F'^es 5-12. This position involved 
live-iw arrangement in family situation. 

1968- 70 Teacher, Research-Child Center, St. Louis, 

(Clinic for emotionally disturbed children.) 
Responsibilities included teaching a group of 
12 emotionally disturbed boys, ages 5-13. 
Research work included work with token 
economies and hyperkinetic children. 

1969- 70 Assistant Basketball Coach, University of 

Missouri, St, Louis, responsibilities 

im olvi^d daily coaching duties and scouting. 

During the first two nonths of Mr. Stenner' s intership, he 
wa.'i involved primarily with proposal development and program planning. 
During this time he worked closely with the Council's research director 
and concentrated on the early childhood education area. 

In april of 1960 the Council was informed of the Dallas Inde- 
pendent School District's experiment with performance contracting. 
Mr. Stenner was asked to spend some time in Dallas to learn about the 
school district's efforts toward innovation and to contribute what 
assistance he could as a Council representative. Mr. Stenner described 
the Dallas experience as " the most professionally exciting and chal- 
lenging experience of my life." 

Dallas had just begun to plan a large scale comprehensive 
performance constructing project. Mr, Stenner became heavily involved 
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in this planning effort due primarily to Mr. Rogers Barton, the District's 
Associate SL5)erintendent for Development. Mr. Barton provided 
extensive one-to-one instruction in planning and management techniques 
during the period Mr. Stenner was in Dallas; Mr. Barton's years of 
experience as a university research director and director of development 
of a regional laboratory made Mr, Stenner *s training indeed a unique 
experience, 

Mr, Stenner impressed with his overall ability and his willingness 
to take responsibility, and he was requested by the Dallas Schools, 
and subsequently authorized by the Council, to act as director for 
management support services of the Dallas Guaranteed Student Performance 
and Training Project. The quality of the planning and the subsequent 
develop effort related to this project gave the project and Mr, Stenner 
national exposure. 

The remainder of the internship was spenV fulfilling obligations 
as director of Management Support Services, making presentations on 
performance contracting and educational program auditing, and writing 
on the above topics. 

Following is a list of publications and presentations made by 
Mr. Stenner during internship. These items are indicative of the wide 
exposure and breadth of the experience he received as a research intern 
with the Council. 

--with Hayman, John L., Jr. 
''Television Behaviour of Young Children", 
Educational Television , May, 1970. 

"Education Performance Contracting 

United States Office of Education, Accountability 

Manual, 1971 [in press). 

"Performance Objectives in Education", 

United States Office of Education, Accountability 

Manual.. 1971 Cin press 

--With Webster, William and Eichelberger, Tony R. 

"A cost-Effective Formula for Performance Contracting", 
Journal of Education (in press) . 

--With Hayman, John L., Jr. 
"Student Performance as an Aspect of Accountability in 
Education", (T)avid Bushnell, ed.), A Systems Approach 
to Innovation in Education . Harcourt, Brace and Javano- 
vich. 

--with Waldrip. Donald R* 
"Education Performance Contracting: The Dallas Project", 
Texas Outlook , 1971 (in press). 
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Monograph: "Education Performance Contraction", 
The Council of the Great City Schools, 1971. 

--with Keane, Michael 

^'Education Performance Contracting: Varied Approaches", 
Journa l of Educational Leadership , April, 1971 

"Education Performance Contracting: Some Policy and 
Legal Considerations", submitted for publication. 

"Performance Contracting: Six Operational Models", 
Paper read at the annual meeting of the Ameriain 
Educational Research Association in Ne\v York, N.Y. , 
February, 1971 

"Management Innovations in Education", Paper read at 
the U.S.O.E, Conference on Accountability, Washington, 
. D.C., 1970. 

"Performance Contracting: A General Overview" and 
"Planning a Competitive Performance Contracting 
Project", Papers presented at the Regional Inter-state 
Project Program, Phoenix, Arizona, March, 1971. 

"Perfomiance Contracting:' A Management Tool" 
Paper presented at the American Management Association, 
New York Seminar on Educational Management Toward 
Accountability, March, 1971. ; ^ 

"Accountability: by Public Demand" 

Paper presented at the annual meeting of the American 

Vocational Education Association, New Orleans, 1971 

"Accountability: An Operational Model", "Performance 
Contracting", and "Educational Program Auditing" 
Papers presented at the Michigan State Education 
Agency, State-wide \>?orkshop on Performance Contracting, 

"An Overview of Accountability" 

Paper presented at the Annual Meeting of the Colorado 
Association of School Executives, Denver, Colorado, 
. 1970, 

"Performance Contracting" and "Educational Program 
Auditing" Papers presented at the Philadelphia 
Accountability llbrkshop, 1970. 

"Accountability and American Institutions" 

Paper presented at the annual meeting of the American 

Education Studies Association, Dallas, Texas, 1971, 

Mr, Stenner entered the internship as a person unknown, with 
limited experience in child care and the education of emotionally 
disturbed children. In what can only be described an an incredible 
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feat, he eraerged after one ^ear as a nationallr recognized eij^ert in 
performance contracting. His in^portant contriDution in the per- 
formance contracting area was possible, of course, only because of 
the unique training and experience he received as a Council intern. 
He is now on the staff of the Me tame tries Corporation. 

4. James H. Van Orden 

The fourth Council ip-^em was Jaines H. Van Orden, Immediately 
prior to joining the Council, Kr, Van Orden. had spent a year in the 
educational research doctoral program at the University of. South 
Florida. Before the experience at Stouth Floxida,-.hD\\r^\refc.l| his career 
had been in writing and in journalism. Among his positions had 
been reporter, magazine writer and editor, and college instructor in 
journalism. His experience as a writer and editor, coupled with the 
knowledge of research gained in a year of formal study, seemed to 
make Mr. Van Orden an ideal person to work on the problem of effective 
exchange of research and evaluation information. 



This possibility was discussed at length in interviews with 
Mr, Van Orden, and he indicated a strong interest -in attempting 
the experiment. Prior to his being named to the internship, he was 
asked to prepare a mock-ip of a possible Council publication, which 
he did. The results appeared promising and he was offered and ac- 
cepted the internship position, 

. During his first months at the Council, Mr, Van Orden prepared 
a number of magazine "paste-i;5)s" and collected a large quantity of 
editorial material. The material included articles written by piablic 
school superintendents in major cities, editorial contributions and 
selected pieces from school newsletters sent in b/ member cities, 

' The piiblication, however, never got off the groimd. Perhaps 
because of his lack of experience in an urban setting, Mr. Van Orden 
appeared unable to capture the gist of issues, which would be considered 
important to member SL5)erintendents and their staffs. Under the sup- 
position that a publication not read and considered frivolous was 
worse than no pii)lication at all. Council staff would not agree to 
reporduction and distribution of any of the "paste-iips." 

After a series of dead-ends, it became apparent that the "re- 
search magazine" concept was not viable, at least with available 
personnel. Thinking of the Council staff was adjusted, and Mr. Van 
orden was asked to work with, the concept of special ad. hoc , reports 
designed to cover important projects the Council was vrorHng on. In 
his initial efforts, he prepared a "Council Brochure" describing a 
number of operational programs and a "Title I" summary derived from 
the larger report described previously. The ad. hoc , reports proved 
marginally successful at best. 

After his first seven months with the Council, months which 
were frustrating for everyone concerned, Mr, Van Orden was reassigned 
to work with the Council ^s director of equal opportimiites, largely 
on writing assignments. He spent the reaminder bf his internship 
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year in this type of activity. Mr; Van Orden^s last months with 
the program were thus more productive, though admittedly he was 
concerned with research only to the extent that some o£ the equal 
opportunity projects, had evalmtion components. 

Mr, Van Orden's tenure with the Council would have to be 
termed a failure, both in regard to his personal growth and in regard . 
to objectives of the Council's component of the training program. 
This is in contrast to outcomes attained with the other interns, which 
are described elsewhere in tiiis report. Reasons for the failure 
will be speculated on in a later section, 

Mr. Van Orden is now employed by the New Jersey School Boards 
Association, 

5. Edward IVhitney 

Edward N. Whitney was the first of the five interns to join 
the Council staff* A native of Philadelphia, he had spent his entire 
professional career in the Philadelphia schools. After nine years 
as a classroom teacher, he joined the staff of the Office of Research 
and Evaluation as a Research Assistant. IVo years later he went- to 
Wayne State University as a doctoral candidate in educational psy- 
chology. Mr. IVhitney came to the Council immediately aft^r conpleting 
his course work and preliminary examinations at Wayne State. 

Mr. Whitney was assigned to the area of information exchange, 
and he was to work on more formal information systems in contrast to 
Mr. Van Orden, who was working with a periodic publication concept. 
Mr. Whitney's first assignment was preparation of a paper on. the 
feasibility and parameters of an information exchange. In the course 
of this work, he had the opportunity to talk at. length with Dr. Lee 
Burchinal, director of the National Center for Educational Communication. . 
A related assignment was the wirting of a position paper on the develop- 
ment of an information exchange system for the member school districts, 
which would be coordinated by Council Staff members. This position 
paper included a component which required the writing of a tentative 
funding proposal. Another task assignment, which Mr. Whitney felt 
to be quite valuable, was the opportunity to serve as the interim 
business officer for the entire operj^tion of the Council. 

Mr. Whitney has noted that the internship afforded him oppor- 
tunities for traveling to m0n±)er school systems, and for attending 
conventions, seminars, and workshops. During these various trips 
there were opportunities for discussion with such persons as Dr. 
Jbdney Napier of Tenple University, Dr. Everett Rodger of Michigan 
State University, Dr, Marvin E. Shaw of the University of Florida, 
and Dr. Marvin Dawson of AurbiOTi University, - 

Mr. Whitney notes the perspectives gained in such problem areas 
as major changes in the racial and socio-economic composition of 
public school populations; intransigence on the part of local schools 
toward making cin-ricular and organizational changes; local political 
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realities which affect the operation of most uarban school systems; 
the extent to which school systems do not comnunicate or share ex- 
periences with one another; and under stajiding of tlie needs in the 
research operations of the large schoel systems throughout the nation. 

To quDte from Mr, Whitney's written evaluation of his intern- 
ship experience, 'Vicarious experience acquired from readings in one 
thing; to be in a situation where you have the opportunity to talk 
first hand with superintendents, school board meinbers, directors of 
research offices, and comiminity meiribers from different parts of the 
country is another and has had a most profound impact on me/' 

Mr. IVhitney is now a District Research Associate with the School 
District. of Philadelphia. 



' c. Common Intern Experiences 

Council interns also had a set of common experiences, designed 
to give all group members a broader understanding of problems faced 
by all large city research and evaluation offices, to develop certain 
ijiportant research and evaluation skills, and to develop a national 
perspective on research and evaluation issues. 

The group met regularly with Council staff members and attended 
meetings of representatives from member districts. Interns Calendine, 
Stenner, and Wiitney attended the Council's semi-annual Board of 
Directors meeting in Buffalo, New York, May 6 through 9, 1970, and 
all five interns attended the Board of Directors meeting in Dallas, 
Texas, November 4 thrugh 7, 1970. 

Board of Directors meetings involve discussions by superintendents, 
board of education members, top-level staff, and invited guests, of 
major issues facing the large urban school districts. Various Council 
committees also meet and discuss problems in curriculum and instruction, 
testing, equality of opportunity, managentent, business operations, 
federal funding, research and eA/aluation, and the like. Committees 
consist of an appropriate representative (such as the curriculum 
director) from each memt^er district. These meetings gave interns 
a perspective on national problems which would otherwise have been 
impossible to obtain. They heard detailed disucssion'^ '^mong persons 
actually involved in the field in attenpting to solve proLlems. 

The interns also attended four training sessions dealing directly 
with research and evaluation problems. As previously noted, three of 
these were sessions funded under a separate training grant, and the 
fourth was coaducted by Teachers College as a part of its project 
responsibilities. Descriptions of each of these sessions follow. 



1. WDrkshop on Measur ^ment of Affective Variables 

A workshop for research and evaluation staff members from Great 
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City districts was held ja February^ 1970, in Meniphis, Tennessee, In 
planning meeting^> , the Steering Cojnmittfie of the research directors 
had decided tliat the topic of 'Measuremeiit in the Affective Domain" 
would be valuable to members of their staff. At the workshop itself, 
Dr. David Orr, of Scientific Educational Systems, laid the groundwork 
for discussion by giving a historical background and speaking to the 
theoretical issues in measuring affect. Follow-up to this general 
introduction was provided by Dr. Marvin Shaw of the University of 
Florida, Gainesville., Dr. Shav; considered the general nature of attitude 
formation and change and also gave an oveiview of available measuring 
instruments. Particular enphasis was given to various problems 
identified by the participants themselves and on areas of concern in 
public education. Dr. Everett Rogers of Michigan State University 
discussed specific attitude measuring methods, focus jng mainly on 
the Likert technique and on sociometric analysis. Dr. Bradley Greenberg, 
also of Michigan State University, addressed the group on the semantic 
differential and ]ed the members in development of their own scales • 
After each topic of discussion the large body of participants broke into 
small groups, with an expert in each group, for individualized in- 
struction and informal attention to specific problems. In addition, 
evening program presentations were given by participants on individual 
experiences within their particular school districts in developing 
and using affective instruments* 



2. Workshop on General Evaluation Skills 

A second workshop for research and evaluation staff was held 
in Monmouth, Oregon, at the Oregon College of Education, in September, 
1970. ITiis session was planned and hendled to allow latitude in 
choosing particular skills to be emphasized. 

In planning for the workshop. Dr. Jack Edling, of ' the Teaching- 
Research Division, Oregon State System of Higher Education, met with 
the Council's Research Steering Committee, presented training materials 
already developed by Teaching Research, and suggested an individualized 
training approach which would allow each research director some lati- 
tude in determining which skills he would like his staff members to 
develop. The Steering Committee voted to follow Dr. Edling's suggestion. 

^■""^ At the training session, research and evaluation staff members 
from Council cities began in one of four areas: proposal writing, 
instructional systems.' measurement, and evaluation. Tests were given 
to determine entry level skills, and instruction was highly indivi- 
zed by the staff of Teaching Research, working under the direction 
of Dr. Dale Hamerus and Dr. James Baird. Participants moved at their 
own rates, and if they reached proficiency level in their initial 
area, they were allowed to choose a second area and begin work there. 

Some 30 research and evaltiation staff members, 21 paid for by 
the project" and the remainder sent at the district's expense, attended 
the three-day session. Proficiency tests given by Teaching Research 
confirmed that intended skills were developed and letters from parti- 
cipants to, the Council indicated their satisfaction with the program. 
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3, Workshop on the Systems Approach 



In May, 1970, the research directors of jneiiiber school districts 
met in Vail, Colorado, to discuss the "Systems Vie\vpolnt in Project 
Development/' Dr* Desmond Cook and his associates. Dr. John Skalski 
and Dr* Gregpry Trzebiatowski from Ohio State University, Kere on 
hand to present not only basic systeirs concepts but also the tolls 
and techniques used by a systems manager. Multi-project management 
\vas discussed in addition to single project development and evalua- 
tion. From a Council member city, Dallas, Texas, Dr. Rogers Barton 
provided input on performance contracting. Additional involvement 
of the research directors themselves occurred in several panel discus- 
sions on such topics as: "Implications of Using the Systems Approach 
for the Research Director" and "What Does Systems Analysis Mean 
Relative to Program Development and Evaluation" and "Where Do We Go 
from Here in the'Sdiool Research Operation?" Indicative of the 
active interest in the topic of this seminar was the fact that many 
cities at their own expense sent more than one participant to the 
meeting . 

Further detai] on the three workshops discussed above is pro- 
vided as Appendices B, C, and D. 



4. Special Workshop at Teachers College 

As a part of its project activities. Teachers College of 
Columbia University conducted a special workshop for the Council 
interns the week of September 28 - October 2, 1970. The purpose was 
to acquaint the interns with the evaluation model being developed at 
Columbia and to give them preliminary instruction in use of the model. 
The sessions were really glimpses of the various facets of the 
activities surrounding the evaluation and field-model work being 
carried out by Teachers College. 

A brief schedule of the workshop follows: 

Monday Briefings by McNeil and lanni, schedule of activities 

roughly outlined. 

Videotape of Drug Encounter group working in coiranunity. 

Tuesday Presentation by Mrs. Cordis on various techniques of 
classroom observation. 



Brief discussion of problems of evaluating EEP (Elementary 
Experimental Program) 

Directions given for self -guided tour of neighboring Harlem 
district and visit to PS 188, 



Torn: of outlined Harlem area. 
Wednesday Sat iii on EEP staff meeting. 
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(contdO 



Sat in on EEP meeting with graduate students (research 
associates) • 



Discussion with Dr, Juleo George of urban education 
problems and social systems theory, 

Thursday Discussed proposed project in banking and the techniques 
of videotaping with Mr, Eric Mortensen- Watched a 
classroom film with split-screen and frame-synchronization, 

Friday Met community relations specialist, Spencer Jamison, at 
his Harlem Center and disuc3sed our visit to Harlem 
District, 

The nDst interesting and information sessions were with Eric 
Mortensen, a specialist In group interaction analysis via video-taping 
and television systems. His work in classroom observational tech- 
niques using split screen photography enables study and replay of 
critical incidents in the classroom, teacher behavior , individual 
student behavior, and the total classroom in siinultaneous tijne 
frames to study relationships of behavior. He also uses time-lapse 
photography to keep a perspective on a single day of classroom unit. 

Mortensen is e'^ o involved with the third-largest bank in 
New York in a project identify and describe '^entry-level jobs" in 
banking to develop a curriculum for training; his ideas and model 
closely resemble the Vocational Education Project of the Council, 

On Tuesday, September 20, the interns met in the morning at 
Columbia to talk with Mr, Spencer Janeson to discuss urban school 
problems. They received his guide, "A Look at the Community," and 
after a brief discussion were instructed to walk from Columbia Uni- 
versity along Amsterdam Avenue to Lenox Avenue, then back to Columbia. 
Jameson's guide listed^areas of interest for the interns to be aware 
of as they walked. The areas were: Housing, Agencies, Grocers, 
Laundries, Automobiles, Department Stores, Candy Stores, Eating 
Establishments, Religious Centers, Transportation, Population, Cultural 
Centers, Education, Politics, Professional Organizations, Recreational 
Facilities, Employment and Fraternal Organizations, 

The purpose of the work is summarized in the following sentence 
from Jameson's guide: ''Armed with nothing more than sharp eyes and 
ears and the questions in this guide, educators will be m^re likely 
to receive full impact of the daily environment of many of the re- 
sidents ,** 

The "full impact" of Harlem was felt. The usual abundance of 
garbage and filth filled the streets, although some streets had their 
own litter campaigns and were noticeably cleaner than others. Fire 
hydrants were open and dripping. It was learned that a ";esponsible 
individual opens the hydrants with a special key if enough reisdents 
desire to be sprinkled, A turned-over and gutted car decorated a 
curb, while other cars lined the streets nose-to-rear* 
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Eyes followed the interns as they progressed up one street and 
dovni the other, into stores, tip and dowi aisles, and into a schoo.'' . 
P,S. 188 was a fortress of another age; spiked finces and barred 
windows gave it a jail -like appearance. A security guard jnet the 
interns and directed them to tJie office. Unfortunately, a tour of 
the facilities did not mteria,lize. 

Later in the walk the interns came across an office representing 
the "store-front" school program. They learned that drop-outs were 
given an opportunity to complete their educations at nearby offices 
and stores staffed by professional teachers. 

The walk ended and the interns compared their irnpressions in 
order to prepare for a debriefing session with Mr. Jameson on Friday, 
October 2nd. During this session the observation guide was used 
and most of the areas were discussed, with a heavy ennhasis on Education. 
Mr. Jameson described the conditions of Harlems' schools, ais well as 
his own program to help youngsters find worthwhile activities and jobs. 
The session ended on the note that educational research is vitally needed 
in the urban school in order to identif>^ problems and help find solu- 
tions to these problems. 

The interns felt that, other than for the activities discussed 
above, the Teachers College workshop was not as well organized as 
it shoiiLd have been. It was of moderate effectiveness* Insight into 
urban problems and into difficulties of conducting evaluation in the 
urban setting was developed, though less than anticipated was learned 
about the new evaluation mo'del being developed. 



d. Evaluation of the Council Component 

Generally, the Council component of the Research Training Project 
can be considered successful. All of the objectives stated previously 
were at least partially met. There was extended study of special 
problems of research and evaluation in large city school systems, assis- 
tance in finding solutions to some of these problems was provided, 
the research capability within the Council was upgraded, and groups 
of interns was provided experiences which, for four of the five at 
least, greatly upgraded their capabilities to engage in research and 
evaluation activities in large urban school districts. The current 
activities of the former interns, the more responsible positions they 
entered after conpleting the program, and their own coments about 
their year with the Council attest to the program's success. 

The Council component was unusual in that it allowed each intern 
to concentrate on his own special interest area. It was highly in- 
dividualized, in the literal sense of the term. Group activities 
accounted for approximately 15 percent of the interns 'time; the re- 
maining 85 percent, each intern was working in a separate area. 
Achieving success in this situation is obviously quite different from 
achieving it when each trainee has a set of well defined courses. 
The Council program required individuals who were mature, who already 
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possessed basic research, and evaluation slcill-s, and who were capable 
of taking responsibility and working independ'3ntly. 

The concensus o£ Cotmcil staff is th?.t the program was highly 
successful with four of the interns particularly so for interns 
Denton and Stenner. It has to be regarded as a failure v/ith Mr. Van 
Orden, and reasons for this failure with Mr. Van Orden, and reasons 
for this failure need to be at least speculated on. 

In the first place, it seems evident in retrospect that the 
selection mechanism was faulty. Mr, Van Orden' s background was in 
i^rriting, editing, and teaching journalism, and he had achieved suc- 
cess in those areas • He had practically no background in education 
nor in urban problems, and he had no field experience in research and 
evaluation. It is evident that some of the background he needed to 
succeed in the task he was given was missing. This is no fault of 
his, of course; it reflects, rather, an error in judgment by those 
who selected him, particularly the project director. This aspect 
of the failure is valuable as regard:5 planning of future training 
activities of this type; it demonstrates the importance of the urban 
education- -research and evaluation experience. 

In the second place, again from the vantage of retrospect, 
Mr. Van Orden was probably not given the close supervision he needed 
under the circumstances. Since experienced persons were to be re- 
cruited, it was expected airing Council staff that interns would be 
capable of taking major responsibilities with a minimum of orientation. 
This mind set was so strong that it required several months for the 
fact that the expectation had not been met in one of the five interns 
to sink in. The delay in this realization was also probably related 
to the success of the other four interns. 

Using the program output the status of the intems at the 
program's conclusion as a criterion, the program can be judged 
about 80 percent successful. It demonstrates the efficacy of the 
training concept being tested, that is, of upgrading the skills and 
operating capability of persons who already have considerable back- 
ground in research and evaluation. It also demonstrates the extreme 
importance of the selection process in this type of situation. Finally, 
it demonstrates the capability of the Council of the Great City Schools 
to handle training of this type, and it emphasizes the value of the 
experiences available through the large urban school districts which 
comprise the Council ^s membership. 



III. PROJECT ACTIVITIES OF 
THE SCHOOL DISTRICT OF PHILAPELPHIA 



a. Background of tixe Philadelphia Component 

Tae School District of Philadelphia^ as one means of helping to 
deal wirth the critical problems it faces ^ hixs upgraded its research and 
evaluation capabilities. Other large city school systems have made 
sijnilar efforts. In order to effect thi:-^ upgrading, personnel with high- 
ly specialized training are needed. This need for personnel has develop- 
ed at a time when there is an acute shortage of qualified educational 
researchers. Moreover, school-system based research personnel require 
skills somewhat different from tho traditional university-based resear- 
cher. They must, have both - sound technical background in methodology 
and the necessary skills and e^q^erience to work successfully within a 
school -system setting. 

The school -system-based researcher (or evaluator, if one prefers 
that terminology) must be able to: 

1. Help educators define their instructional objectives. 

2. Design studies which will yield the desired information 
without distorting or too heavily intruding upon the 
programs being studied. 

3. Select or design in5truments which are appropriate for 
the population being studied, 

4. Analyze and interpret the results of studies. 

5. Present the results in such terms that they will be of 
maximum immediate use to decision makers. 

Personnel with these skills are in short supply. By becoming in- 
volved in the training of research personnel, school- system research 
offices can not only help supply this need, but can also add the vital 
ingredient of familiarity with the school situation which is lacking in 
traditional programs of preparation. Thus they can produce personnel 
who are capable of research which is not only technically sound, but 
also relevant to the needs of the schools. 

It is \\Xth this motive in mind t?iat the Office of Research and 
Evaluation of the School District of Philadelphia undertC/ok to develop 
a Research Training Program. The program had these objectives: 

a. To develop, through a combination of practical e:xperience 
and formal training, an overall capability for condi ting 
hi.(?h qiiality research and evaluation in school systems. 

To develop, through a combination of discussion and direct 
experience, an imder standing of special problems and special 
needs related to conducting research and evaluation in the 
school setting. Among matters receiving special attention 
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win be; determination of school issues which are amenable 
to research solution; stating relevant research objectives; 
interpersonal relationships with professional school person- 
nel; regidLar project iiionitoring and feedback; and reporting 
in an understandable way to decision makers • 

c. To provide the practical competence to. conduct research 
and evaluation in schools through on-the-job experiences 
which deal >vith special problems of design, planning, con- 
trol, and conduct of research projects; measiirement and 

. retrieval; statistical analysis of data; and interpreta- 
tion and reporting, 

d. To provide the teclmical competence needed to conduct educa- 
tional research, and evaluation throiigh formal graduate- level 
university course?^ in ^^'^arch planning, research methods, 
tests and measurements, statistical analysis, and psychologi- 
cal learning theory. 

e. To develop, through discussion and on-the-job experience, 
the ability to help school. personnel identify, clarify, and 
phrase their instructional and administrative objectives. 

f . To develop, through practical experience and understanding 
of the ways research and evaluation study results shotild 
be reported in school systems so that they have maximum im- 
pact on the decison-making process. 

g. To develop in trainees a comiiritment to school research and 
evaluation as a career field and to assist them in locating 
a suitable position after the program is completed. 

b. Operation of the Philadelphia Component 

Approval of the project was arixirjunced late in the Spring of 1969, 
about the time most colleges and universities were completing their 
spring terms. Since the trainees were to enter the program the follow- 
ing September, recruiting became a major problem. The purpose of the 
recruiting effort was to identify six college graduates who were: 

. Capable of pursuing a masters degree in Educational Research. 

. Capable of practicing in a large school system research office. 

The recruiting effort was conducted in three phases: Advertising, 
Screening, and Interviewing. 

Durin-; the first phase, circulars were distributed both within the 
School System and to colleges and universities. Letters were sent to 
university placement officers, as well as the chairmen of the inathema- 
tics, psychology, and education departments. Dean Gross of the 
University of Pennsylvania contacted hi? colleagues ::Ji a number of 
Eastern Universities. The Office of Infoimational Services of the School 
District of Philadelphia prepared press releases. 
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In all, over one hundred inquiries were received concerning the six 
positions available. About forty individuals .con^leted applications 
and were assisted in applying for admission to the University of 
Pennsylvania.. 

The second phase of recruiting involved screening by the University 
of Pennsylvania. Only those applicants accepted by the University were 
interviewed. Nineteen interviews were conducted, and the six trainees 
were- selected from this groip. 

The recruiting experience brought to light only one major problem. 
Notice of Grant Award from U.S.O.E. came so late in the year that most 
colleges had already dismissed for the siBimer. This had the effect of 
seriously restricting the pool from \>tfaich the program could draw. Most 
top students had already received offers of schooling or enployment. 
Only those students remained \dao had not received offers or acceptances 
or those who had not decided what they wished to do. Even under these 
circumstances, the program was able to identify and attract a number 
of promising trainees. 

Notification of grants earlier in the year woiild, however, permit 
selection from a larger pool. The program would then have been able 
to conpete for the top students. 

2. The. University Subconponent 

Formal Coursework. Trainees in the program were enrolled as 

graduate students at the University of Pennsylvania and attended cour- 
ses on a half-time basis. The function of the Master of Education Pro- 
gram was to equip trainees with the technical skills necessary for 
conducting competent research. 

Trainees were a part of the METER (Measurement, Evaluation, and 
Techniques of Educational Research) program under the chairmanship of 
Dr. Frederick B. Davis. Professor Davis and Dr. Andrew Baggaley shared 
the responsibility of advising the trainees regarding thier academic 
programs . 

During each of the four semester of the Research Training Program, 
trainees took two course units (six semester hours) of work at the 
University, and during the intervening simmer an additional three course 
units were taken. Courses taken were basically in the area of Educa- 
tional Psychology, Educational Measurement, Research Design, Test 
Construction, and Statistics. Some electives were permitted. 

Analysis of the Formal Coiirsewbrk. TfUs phase of the program 

was satisfactorily planned and executed. The general pattern of courses 
was agreed upon in advance. Dr. Davis and his staff proved very flexi- 
ble in developing a program with each trainee. 

The Summer Workshop. -- During the second half of the summer 
intervening the two years of the program, tiainees participated in an 
all-day workshop. liie workshop was designed to: 
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• Give trainees the experience of carrying out an indepen- 
dent research project, 

, Give trainees the opportunity to apply skills learned 
during the first year. 

. Serve as a vehicle for teaching additional skills. 

The summer workshop was divided into two portions. During the 
morning session trainees were instructed in mss data processing and 
during the afternoon they worked on individual projects. The instruc- 
tors i^^TVolved atteriipted to integrate the two experiences. An over- 
view of the workshop can be found in Appendix H, 

Since the workshop was developed specifically for this program, a 
few words of explanation might be in order. The inoming session was 
devoted to mass data processing. Trainees were introduced to conq)Uter 
systems, a programming language (FORTRAN , and packaged programs. 
Mien the data from their individual projects was ready for reduction 
and analysis, it was used as a vehicle for instruction. A conplete 
course outline can be found in Appendix I. 

During the afternoons, trainees worked on individual projects. In 
order to simulate as much as possible real working conditions, projects 
were assigned to trainees. (TProjects were selected by the course in- 
structor in consultation with the trainee advisor) . Each trainee then 
had the responsibility of writing a proposal. He was then supplied 
with a set of appropriate data and asked to reduce and analyze it. A 
final report was then written. During the course, the instructors 
served as consultants to the trainees on an individual basis. Several 
days a week the class met as a group for more formal instruction and 
discussion. A course outline will be found in Appendix J. 

A nalysis of the Summer Workshop > -- It was this portion of the 
program that was most enthusiastically received. After having spent a 
year working on parts of studies and taking formal classes, trainees 
were finally given an opportunity to "put it all together" in a pro- 
ject of their own. Trainees reported that they had gained a lot from 
the workshop. (Here, as in the other portion of the University com- 
ponent, no measure of trainee learning was made beyond the report of 
the University) . 

Since the workshop was designed specifically for this program, the 
instructors observed carefully the outcomes of their efforts. At the 
end of the workshop, each instructor was asked to detail the skills 
vAiich trainees could reasonably be expected to display after the work- 
shop. In addition, they were asked to critique the workshop as they 
had conducted it and irvke recommendations for revisions. 

The reaction of the instructor of the mass data processing portion 
of the workshop can be found in Appendix K, 

The reaction of the instructor of the individual project portion 
of the workshop can be found in Appendix L. 
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On- The - Job- Tr a jjiing . This part of the Research Training Pro- 

grm Wcis ijxtended to supply pr^-ctical e^erience for the trainees. 
By^ assistijig a staff jneniber xn his wrk, the trainee "would be able to 
experience the probleins of conducting research in large city school 
systems. He vould be able to bring the practical probleins met to his 
course work, and, similarly, bring the information gained from course 
vrork to his job. 

In addition, trainees would become familiar with the workings of 
a large city school system and of the informational needs of educational 
decision makers (both teachers and administrators) . 

Trainees were assigned to each of the three Divisions of the Re- 
search office for a period of one semester. Each trainee was given 
experience in Division of: 

Instructional Research and Development 
. Testing 

, Administrative and Survey Research 

Durinf^ his assignment in a Division, the trainee worked half time 
(the other hald of his day was spent at the University) either as an 
assistant to a Research Associate or on a project assigned directly by 
the Division Director. 

Trainees assisted in such tasks as: 

collecting data 
onducting literature reviews 
, ^ eparing testing reports 
, w. iting research proposals 
. writing research reports 

conducting opinion sun^eys 

conducting pupil population studies 
, conducting studies of pupil accounting forms 
. monitoring programs being studied 

A senior staff member was appointed to serve as Program Director 
and Tainee Advisor, He had the responsibility of coordinating the pro- 
gram, acting as liaison with the University, conducting the seminar, 
and advising trainees in their adjiostment to the office and to the program* 

Analysis of On-The-Job-Traiaing, This part of the program 

gave rise to one major problem. In an attenpt to introduce trainees to 
all of the functions of the Research Office, they were assigned to 
each Division for one semester. IVhile this permitted a breath of ex- 
perience, it militated against depth. Trainees started a project in 
one Division but had to leave before it was completed. They ten ro- 
tated to the next Division to complete a project they had never started* 
This produced a certain amount of frustration among the trainees. 

Though a research director in a small town school system might 
have to be a "jack-of-all -trades," large city school system research 
offices house a number of specialties. There is a certain arount of 
overlap of skills from Division to Division, but there are also 
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some skills that are specific to each Division. It might have been 
better to recognize this fact and require each trainee to specialize 
in one Division. Fajniliarization with the functions of other Divisions 
could then have been accomplished by a series of seminars. 

Seminar Program. As a part of their experience within the Office 
of Research and Evaluation, trainees attended a series of specially 
designed seminars. The seminars had the function ofi 

, giving the trainee a feeling of identity with the program. 

. providing a forum for the discussion of problems, 

, providing a vehicle for feedback from trainees. 

. giving trainees an opportunity to explore additional 
areas of interest. 

Seminars were held every other week during the first semester of 
the prograjn. They were two hours in length. Three general areas of 
content were included: 

1) Orientation 

Discussions dealt with the organization and operation 
of the Research Office and of the school system in 
general. Topics included administrative-organization, 
history, problems, operation, etc. 

2) Problem Solving 

Trainees were encouraged to discuss problems relating 
both to personal orientation to the Research Office 
and to issues specific to individual assignments. 
Solutions were proposed, and, where necessary, policy 
decisions /ere made. 

3) Planning 

Trainees were- involved in two kinds of planning: 

for specific events (i.e., the AERA convention., 
visitation to Early Childhood Programs). 

for future seminars (Trainees proposed areas of 
interest, speakers, etc.) 

During the second semester, a more structured pattern was intro- 
duced. Specific topics were cov^rred. These were based both on the 
requests of the trainees and on additional material injected by thie 
Trainee Advisor. 

When speakers were invited, one trainee accepted the responsibility 
of meetijig with him in advance to orient him to the concerns of the 
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of the trainees. The same trainee then acted as chairman for the seminar 
session. 

Trainees elected to meet on a weekly basis during the second semes- 
ter. 

Topics included: 

1) Behavioral Objectives 

Discussion included rationale, stiaicture and practi- 
cal applications. Trainees wrote sample objectives and 
discussed them. Each trainee had read Preparing Instruc- 
tional Objectives by Mager before the seminar. 

2) The Testing Program in the Philadelphia School System 
(2 sessions) Speaker: Mr. Jules Grosswald, Dirr;^::or 
of Testing. 

These sessions provided a comprehensive overview of 
the school system's testing program, including purposes, 
instruments, and problems. 

3) The Role of Educational Research in the School System 

This session used The Saber -Tooth Curriculum by 
Harold Benjamin as a springboard. Trainees read the 
book prior to the session. Each trainee presented a short 
paper an his perception of the role of research in the 
school system. Papers were discussed and critiqued by 
the group. 

4) Early Childhood Programs 

Mr. Milton Goldberg, Director of Early Childhood 
Programs spoke on the various programs currently in 
operation in the school system. 

5) Program for Advanced Administrative Development 

Mr. McKinley Lennox, ^valuator attached to this pro- 
gram, discussed his role as an evaluator during the 
fomative stages of a program. He stressed reD.ations 
with participants, coJ lection of data, and development 
of objectives. 

6) Bilingual Program 

Mr. Robert Offenberg, Program Evaluator, discussed the 
evaluation being carried out for this program. He stressed 
design and planning, relationships with participants, in- 
struments used, and problems encountered. 

7) The Role of the Curriculum Office in the School System 
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Dr. I. Ezra Staples, Associate Superintendent for 
Instructional Services, spoke about the functions of the 
curriculum office. He stressed the relationship betv>reen 
the curriculum office and the schools. 

8) The Role of ResearcJi and Evalviation in the School System 

The starting point for discussion was Educational 
Evaluation: New Roles, New Means (N.S.S.E. 68ui Year- 
book. Part 11] . The first session dealt with the re- 
lationship between traditional research designs and 
current thinking in the field of evaluation. Several 
subsequent sessions dealt with specific issues. 

The seminars were reinstituted during the third semester (second 
academic year) of the program, and they included a visit to Research 
for Better Schools, the Regional Educational Laboratory. Toward the 
end of the third semester, the seminars were suspended for the remainder 
of the year. 

Pressure from increased responsibility on the job, more difficult 
course work, and anticipating preparation for masters comprehensives led 
trainees to resist taking on additional assignments. As conceived, the 
remainder of the second year seminars were to be run largely by trainees, 
and extensive preparation would have been necessary. 

Analysis of the Seminar . The seminar appeared to have served 

well its initial purpose of giving trainees a sense of identity as a 
group. University personnel commented on their cohesion, and even after 
the program ended there was still some contact among them (i.e., they 
seemed to know what the others were doing.) 

During the first year the seminar was used as a forum for comments 
on or grievances about work in the Research Office. When required, ac- 
tion was initiated as a result of such discussions. 

The effect of the seminar in the cognitive area is not as clear as 
in the social area. Topics were, for the most part, chosen in consulta- 
tion with the trainees. An attempt was made to respond to needs and 
interests as they developed. 

Topics covered are felt to have been pertinent and worthwhile. The 
seminar might have been improved, however, if greater structure had been 
introduced somewhat earlier. This would have permitted, for example, 
more extensive coverage of the expanding literature in the field of 
evaluation techniques. 

Another consideration is the format of the seminar. Probably more 
reliance should have been placed on retaining consultants (including 
specialists from outside the school system) to speak to the trainees and 
conduct discussions. In contrast, when the seminar started to function 
as an additional academic component there was resistance to it. It was 
not able to be a classroom, and it might well have been a mistake to 
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attempt to make it evolve into one. 



Independent Projects . During the fourth semester of the program, 

trainees worked on independent projects. This coarrqDonent of the program 
was included to: 

. give trainees the opportunity to conduct individual studies. 

. help trainees become aware of deficiencies in the backgrounds. 

. provide an additional experience in a practi-zal way. 

Before the end of the third semester, trainees were asked to select 
independent projects. The following ground rules were established: 

. The project had to be one the trainee could complete before 
the end of the year. 

The project had to be r benefit to the School District. 

. The trainee had to find a "sponsor" who would guide him 
in the completion of the project. 

Beyond the above, no restraints were placed on trainees aud tliey 
were free to pursue their own interests. 

Trainees were required to submit brief proposals for their projects. 
When these were approved, they were permitted to begin working on them. 
The following projects were undertaken: 

. Evaluation of Parent -Aide Component of the Project for 
Improving Reading -Language Teaching (PIRLT) 

. A Study of the National Teachers Examination as a Criterion 
for Teacher Placement 

. An Evaluation of a Reading Teacher Consultant Program for 
New Teachers. 

. An Examination of the Use of Research Information by 
Teachers Under T^:;d Conditions of Support 

Analysis of the Independent Projects . All four projects under- 

taken (There were only four projects because one trainee graduated early 
and two others resigned. This will be discussed under the Progress 
Toward Objectives section, below.) were successfully completed. 

Two (the NTE study and the PIRLT Study) were aspects on ongoing pro- 
jects, and the Research Associates responsible for the studies served 
as sponsors. The study of Teacher Use of Research Information was per- 
formed at an elementary school under the sponsorship of the principal, 
a former Research Associate in this office. The study of the Reading 
Teacher Consultant Program was performed under the sponsorship of a 
District Research Associate. 
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The major problem observed in this component of the program resul- 
ted from the freedom given to trainees in project selection. A lot of 
time ^^ras lost casting about for projects, and few got underway before 
the end of January or the beginning of February. A better plan would 
have been the direct assignment of projects based on existing needs of 
the office and knowledge of trainees' interests. 

C. Evaluation of the Philadelphia Component 

Meeting of Objectives . In terms of progress toward meeting ob- 

jectives, the Philadelphia component can be evaluated as follows: 

Objective 1. To develop, tlirough a combination of practical 
. experience and formal training, an overall 
capability for conducting high quality research 
and evaluation in school systems. 

No assessments of knowledge - in the paper and pencil test sense - 
were made beyond those of the University. All graduates of the program 
were judged, by Research Office leadership personnel, to be capable 
of occupying at least Research Assistant positions. Additional experi- 
ence and some progress toward the Doctorate would be required for Research 
Associate level positions in this office. 

More concrete evidence along these lines is the fact that no trainees 
have been hired as Research Assistants in this office, and another trainee 
has been offered a Research Associate position in another chool system. 

Objective 2. To develop, through a combination of discussion 

and direct experience, an understanding of special 
problems and special needs relating to conducting 
research and evaluation in the school setting. 

Among matters receiving special attention will be: 
determination of school issues which are amenable 
to research solution; stating relevant research 
objectives; interpersonal relationships with pro- 
fessional school personnel; regular project 
monitoring and feedback; and reporting in an 
understandable way to decision makers. 

No formal assessment of knowledge in this area has been made. It 
should be noted, however, that seminar sessions, on -the -job -training ex- 
periences, and total office staff development seminars ^^^ere directed to- 
ward this objective. 

Objective 3. To provide the practical competence to conduct 
research and evaluation in schools through on- 
the-job experiences which deal with special 
problems of design, planning, control, and con- 
duct of research projects; measurement and data 
retrieval; statistical analysis of data; and 
interpretation and reporting. 
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In addition to on-the-job training experiences, both the Summer 
Workshop and the Independent Project were directed toward this objective. 
Both program con^onents required trainees to deal v/ith these problems 
and to produce products which demonstrated acceptable solutions to them. 

Objective 4. To provide the technical competence needed to 
conduct educational research and evaluation 
through formal graduate -level univeristy courses 
in research planning, research methods, tests 
and measurements, statistical analysis, and 
psychological leamijig theory. 

Four trainees received their Master's Degrees in June, 1971. 

One trainee dropped out of the program at the end of the 
year for academic and personal reasons. He was replaced 
with another trainee, but the replacement does not have 
sufficient credits for the degree. 

One other trainee dropped out during the second year due 
to a decision to go into another field. 

Objective 5. To develop, through discussion and on-the-job 

experience, the ability to help school personnel 
identify, clarify, and phrase their instructional 
and administrative objectives. 

The The objective was pursued in the seminar, the summer workshop, and 
in the independent project, as well as in the work- experience component 
of the program. 

Objective 6. To develop, through practical experience an 

understanding of the ways research and evalua- 
tion study results should be reported in school 
systems so that they have maximum impact on the 
decision -making process* 

Progress toward this goal was made by having trainees participate in 
writing research reports, as well as in the summer workshop and the inde- 
pendent project. Realistically, this goal is one toward which all re-' 
search offices are working, and no one can claim to have "arrived.'^ 
Trainees did, however, participate in some of the efforts of this office. 

Objective 7. To develop in trainees a commitment to school 
research and evaluation as a career field and 
to assist them in locating a suitable position 
after the program is completed. 

Seven trainees participated in this program (two fox one year only) . 
Of this number: 

. One left for academic and personal reasons. 

. One left to go into another field. 
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. One is to work as a Research Associate in another 
school system. 

. One has been accepted in a doctoral program in 
Educational Research. 

. One has been accepted in a doctoral program in 
Educational Administration. 

Two have been .hired as Research* Assistants in this 
office. 

For the short run, then, the program has been successful in four 
out of seven cases. One is in a related field, and two were outright 
failures . 

Conclusions and Recommendations . Overall, this approach to the 

training of researchers for large city school systems appears to have 
realized its expectations. Trainees were able to earn graduate degrees 
in Educational Research and Measurement, and they did have the experience 
of coping with problems of doing research in a school system. These ob- 
servations, however, are of the short range. No statement can be made 
about how the trainees will develop. Moreover, replications beyond the 
specific trainees involved and beyond the specific school system involved 
are necessary before meaningful generalizations can be made. 

Before replication is attempted, two issues should be given consi- 
deration. First, the present prograin is a relatively extensive one 
and probably is not feasible for school systems in its present form 
without outside support. Even if school systems are ible to afford the 
salaries of the trainees, it is unlikely that the/ would be able to af- 
ford the tuition as well. That expense would have to be bomo by an 
outside source or by the trainee himself. 

Another consideration is staff time. Working with a trainee requires 
a certain amount of tiine, and this must be taken into consideration when 
distributing responsibilities to staff members. Unless some advance 
thought is given to the role that a trainee is to play in a division or 
in a project, there is a great danger that he will be given mostly cleri- 
cal type tasks to do. This, of course, would defeat the purpose of the 
work- training component.^ 

The following recommendations are reiterated from earlier discussions 
of program components: 

1. Notice of grant award should be given no later than 
January or February. This will permit the program 
to solicit participants from a much larger pool than 
was available to the cuirent program. While there is 
no gioaranteem there is at least a probability that 
some of the individual failures of this program could 
have been averted if selection would not have been 
limited to students still uncommitted in May. 
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2. The University programs o£ traiaees should be so 
planned that all have at least one course in FORTRAN, 
one course in intermediate statistics, and one course 
in research design prior to the workshop. This will 
permit trainees to get maximum benefit from the work- 
shop experiences. 

3. Trainees should be assigned to one division for the 
entire program. Familiarity with other Research 
Office functions should be achieved through seminar 
sessions and through the natural interface of the 
divisions. This will avoid fragmented experiences 
and permit trainees to gain more in-depth knowledge. 

4. The seminar program should be modified so that the 
latter part of the first year and the entire second 
year involve more meetings with specialists, both 
from within the school system, and from outside 
sources . 

5. The independent project should be assigned to the 
trainees and he should then be assigned to a sponsor. 
This would not only more faithfully duplicate actual 
working conditions, but also would avoid the time 
lost casting about for a problem. 
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IV- PROJECT ACTIVITIES OF 
TEACHERS COLLEGE, COLUMBIA UNIVERSITY* 



a. Background and Rationale 



This section of the report describes the training model de- 
veloped by Teachers College, The inodel develops out of a comprehensive 
survey of materials on evaluation and qualitative research, and out 
of our own recent experience with several large-scale evaluations. 
We would call attention to several important features of this report: 

1. It is concerned with training evaluators in the 
broadest sense of the term "evaluation." We 
believe that tlie evaluator must understand how the 
evaluative function serves decision-making func- 
tions, and also how evaluation is related to issues 
of valuing, determining worth, and making value 
judgmen:s. 

2. Given i primitive state of the art in edix:ational 
research, we believe that fieldwork and qualitative 
analysis are critical features of an evaluation. 

We have been experimenting with some of these tech- 
niques in our own work, and are confirming are own 
hunches as to their usefulness. This report pre- 
sents an introductory course for training students 
to carry out fieldwork and qualitative analysis. 
"Learning by doing" is the basic training methodology. 
To our knowledge, the program outlined here is the 
only attempt that has been made to systematize 
training in this area. We feel that is a significant 
contribution- -all the more so because it can be 
used as ah introduction to social science methodology 
as well as an introduction to evaluation, 

3. The program outlined here is flexible enough that 
the introductory portions may be conducted by a 
non-acade].nc institution such as the Council or 

a Board of Education; introductory portions are 
equally appropriate for use in academic institutions. 
Pfowever, more advanced training would probalby 
have to be provided primarily through the academic 
institution. 

4. Internship as a membe^" of a team, conducting an 
ongoing evaluation, is a key element of this program. 

^The mterial in this section , is taken ver batem in a number of instances 
from the report submitted to the Council by Teachers College. The 
editorial 'Ve," which appears quoted often, represents the Teachers 
College staff and not the Council. 
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We believe that there are Jnany thiixgs about school 
sy-stems that can not be taught without direct and 
continuing contact vrith such systejns. The internship 
provides for this kind of contact, tinder the guidance 
of a more experienced team member. It also provides 
the interm with e:xperience in working as part of 
a team- -an experience v^ich should prove valuable for 
later professional partic ipation as a team member. 

5. We have tried to suggest ways in viiich our training 
outline can be related to the work of others who 
are also concerned with training. The v/ork of Malcolm 
Provus and of the Center for the Study of Evaluation 
are of course notable in this regard. 



1. :^^eneral Characteristics of the Proposed Evaluation Design 

and Training Morsel 

Evaluation, Administration and Decision Making Evalioation 
is commonly described as an aid to decision-m2.king and administration. 
Both evaluation and effective administration involve a thorough 
analysis of program goals and of activities which purport to meet 
those goals. Effective evaluation and effective administration are 
mutually , inextricably, dependent . 

*Yet there is a persistent tendency for evaluation to be 
dealt with as a discrete activity- -in government contracts, in academic 
course, and in evaluation literature. Contracts call for an "outside" 
evaluator, and too often that evaluator is "called in" after a program 
has already been shaped by its administrators. An implicit assumption 
throughout so much of the literature is that evaluation is something 
imposed onto a program; the evaluator is viewed in effect as a tech- 
nician- -as one who uses his technical skills to determine the success 
of a program in measuring output. 

We can speculate that the main reason for this is that it is 
convenient and sometimes even necessary, for conceptual purposes, to 
separate the function of evaluation from other administrative func- 
tions. Evaluation was often considered "research" as opposed to 
"action;" as "research" it had to follow the canons of non-intervention, 
usually resulting in non relevant behavior. However, our own experience, 
as well as that of some others, demonstrates the importance of grounding 
the evaluative function in the process of administration. In tliis 
way, program goals can derive from a comprehensive perspect, they 
can be operationalized in such a way as to aid program decisions, and 
value judgments underlying administrative choices can be made explicit. 

For these reasons we feel it is extremely important, in the 
training of evaluators, t3 stress the interdependence of evaluation 
and administration. Here at Teachers College we are in fact de- 
veloping courses and experiences in evaluation to be included in the 
program for training of educational administrators. The courses and 
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e:)^erlences in evaluation to be included in the program for training 
of educational administrators. The courses and e;?qperiences in 
evaluation described in the next chapter? of this report, are typical 
of those that might be included in that training progx^Rjn. While the 
courses and experiences might also be taught as a descrete sequence, 
we feel it would be better to integrate them with a broader program 
for the training of administrator-evaluators or program developers. 

Multi-Faceted Projects; Impdrtarice of a Team Approach The 
kinds of problems requiring evaluation in education are so varied 
and often so complex that they require a wide range of evaluative 
cQiT^Detencies. I*) single person can embody that full range, and for 
this reason we feel that a team-approach to evaluation is critical. 
The teamsshould consist of a number of individuals each of whom in- 
corporates one or several of the special competencies required for 
the specific problem under investigation. Each member of ' that team 
should also have a general knowledge of the field of evaluation and 
should be able to deal conceptually- -if now technically- -with the 
problem that the team as a whole is analyzing. 

We feel that trainees in evaluation, who will ultimately be 
members of such a team, need to have some experience in a team re- 
lationship prior to assuming full responsibility as a team member. 
For this reason, we suggest that the trainees program include an 
internship experience with a team. Such an internship experience 
has been provi^ded for selected students at Teachers College over the 
past two years. Our experience indicates that this is indeed an 
effective training device. 

Expected Careers for Trainees; Relevance of this Program --As 
noted the concern is to train evaluative personnel who understand 
their function as integral to, and can operate within, a broader con- 
text of administration and decision makings In fact we would antici- 
pate that some individuals trained in this manner might mve into 
high level administrative positions for innovative programs where 
eA^luative competence is critical. The most obvious type of position 
falling within this category is of course that of director for 
research anc evaluation in a school system. The, Council of the Great 
City Schools is in a position ot help place such individuals, and 
plans by which this might be acconplished are included elsewhere 
in this report. For these kinds of positions, it is important that 
the individual have a comprehensive understanding of evaluative 
issues, and that he be able to conceptualize particular evaluations 
in a way that permits him an overall view of the evaluation while 
others focus more carefully on specific segments of the study. 

Others m our program might wish to confine themselves more 
exclusively to evaluation per se . Among these are individuals vJxo 
may wish to 1) direct or conduct evaluative consultation services or 
2) provide specialized or technical service in condixiting portions 
of evaluation studies which requires their specific skills --specialties 
might include classroom observation, psychometric testing^ computer 
technology, etc. No doubt some would wish to locate at a university 
and to teach courses in conjunction with their particular intersts in 
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evaluation. 

All can be trained from the type of program outlined below. 
The program provides for 1) a comprehensive overview of issues relevant 
to evaluation; 2) practical experience in carrying out an evaluation- - 
acconpanied by a seminar to facilitate conceptualization and provide 
feedback and 3) in depth training in one" or several specialty areas. 
Concentration among the taree areas would vary depending on trainees' 
ultimate goals. 

Implementation of the Program in Academic and Non -Academic 
Settings The preceding section outlined possible careers for 
evaluative personnel: it can be anticipated that each type of career 
requires a graduate degree Cs) for advancement in that career. There- 
fore the discussion that follows is oriented toward graduate training- 

However, two major components of the program outlined here-- 
1) internship in an ongoing evaluation and 2) participation in a 
core curriculum- -can be administered by a non-academic institution 
such as the Research Council. If such were the case, that institution 
to provide additonal training which would build onto the core and 
internship experience. 

The core and internship experience can also be integrated 
into college or university degree granting programs. The conponents 
are of course basic to any program leading specifically to a degree in 
evaluation. However, if such programs are not in existence at a 
particular institution, the components can be linked to other kinds 
of degree programs --such as administration or curriculum i.o provide 
appropriate training in evaluation. 

Critical to our program is the student's internship e:}q5erience 
as a team member of an ongoing eA^luation; the existence of an on- 
going team evaluation is thus a prerequisite for this program. For 
the past two years we have been involved in such an evaluation and 
have experiemented with the possibilities' of attaching students to 
such projects at the same time that they are taking courses or com- 
pleting degree requirements. As expected, we experienced some dif- 
ficulties and are planning certain program revisions. But we feel 
that basically our approach to the internship experience is in fact 
a viable one. 

Given a multi-faceted ongoing project of the kind described 
above, there is a range of types of positions for interns. It is 
possible therefore for interns to enter the program at various stages 
of their educational career. The ideal intern is one who has had 
some experience i.n an educational setting, and who has a strong 
academic backgrouu^d in the social sciences, and perhaps in education. 
Interns just beginning graduate study may assume positions of minimal 
responsibility, imder the close, supervision of another team member. 
Interns at more advanced stages of graduate stydy could assume posi- 
tions requiring wore responsibility. An intern might remain in the 
program for several years, assuming increased responsibility, until 
he is ready to move into one of the careers described in the previous 
section. 
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2. Use o£ the Term '^Evalmtion'^ 



The literatxire o£ the past forty years--arid particularly recent 
literature- -displays a bewildering array o£ definitions for the term 
"evaluation." Rather than quibbling with these definitions and elabo- 
rating yet a new one for the list, we prefer to accept three different 
sets of definitions as appropriate to our interest here. Although 
each set has a somewhat different focus, we fee] that all fall appro- 
priately under the heading ''evaluation.'' The three types of definitions 
are discussed below, and each is touched on in the training program 
vMch follows. 

Evaluation as the study of input-output relationships- -One 
definition focuses on the relationships among the inputs, processes and 
outputs for a particular program. An attempt is made to identify 
and describe the. inputs and processes required to produce certain kinds 
of outputs. By obtaining this kind of infoimation on the relationship 
between means and ends, a planner or decision maker can manipulate 
means variables to achieve desired ends. Evaluative information of 
this sort on past performance is particularly useful in planning for 
and making 'decisions about future performance. It was in this sense 
that Hyman's now classic definition was offered: 



A siniilar definition by Siichman incorporates a range of A^ria- 
tions on the same theme, which he encountered upon reviewing a vast 
literature on evaluation: 



. .we may simply indicate the range of variation 
by defining evaluation as^e determination (whether 
based on opinions, records, subjective or objective 
data) of the results (whether desirable or undesirable 
transient or permanent; immediate or delayed) attained 
by some activity (whether a program, or part of a 
program, a drug or a therapy, an ongoing or one-shot 
approach) designed to accomplish some valued goal or 



Recently, the growing interest in systems approaches to decision 
making has placed a special emphasis or the analysis of alternatives 
within this, means -ends framework. Assuming that there are more than 

IT Herbert I-^onan, Charles Wright, ^id Terence Ffopkins, Applications 
of Methods of Evaluation , Berkeley, University of California Press, 
1962, p. 3. 

2/ Ed^^^ard A, Suchman, Evaluative Researc h, New York Russell Sage 
Foundation, 1967, pp. 31-32. 
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one set of means for achieving some end, analysis fociises on selecting 
the "best*' means. Of the many considerations which enter J.nto a de- 
termination of "best/* that of efficiency is receiving increasing at-* 
tention. The relationship between efficiency (particularly cost benefit) 
considerations and our evaltiation systems framework, is perhaps best 
captured by Alkin who offers the following: 



Evaluation is the process of first identifying and 
then quantifying, or measuring, the relationships 
between students inputs and educational outputs and 
determining the combination of mediating factors 
which maximizes the educational outputs, given a 
constant financial input and controlling for the ef- 



The point here is not that all evaluative studies must fall 
within this tightly structured, quantitative framework, but rather 
that cost benefit issues fall appropriately within the domain of 
evaluation, and furthei' that the issue of efficiency is an important 
area of investigation for the evaluator. 

Evaluation as an aid to decision-making- -A second focus is 
that of providing decision m-ikers with the kinds of information they, 
need to improve the quality and impact of their decisions. Both the 
CIPP model and the model of CSE derive from this perspective. This 
is seen clearly for example in the CSE definition of evaluation: 



From this viewpoint the decision maker decides what areas of information 
he wants, and the evaluator then sets about providing that information. 
Although he may help the decision maker select informational areas, 
the evaluators* prmsLry task, according to this perspective, is merely 
the provision of information. Much of the informatio;, gathered will 
likely be cast into the evaluation, systems framework described above. 
But the e^^luator who services the decision maker is not permitted to 
impose his own system onto the analysis- -he must use that of the 
decision -maker. This frees the study from much of the evaluator 's 
own subjectivity, although it introduces the decision makers subjectivity 
into the analysis. The implications of this will be taken up in the 
next few pages. 

17 Marvin C. Alkin, Towards an Evaluation Model: A Systems Approach 
Los Angeles, Center for the Study of Evaluation of Instructional 
Programs, University of California, December, 1967, p. 1. 

2/ Marvin C. Alkin, ''Behavioral Objective Specifications in Evaluation: 
Relevant or Irrelevant,'' Western Regional Conference in Testing 
Problems, Princeton, Educational Testing Service, May, 1969, p. 2. 
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Both the CIPP and CSE jnodels assert that the decision-maker 
requires a specific kind o£ information for each of the several sequenced 
stages involved in planning and implementing a prograon; the task of 
the evaluator is to stpply that information t CSE identifies the 
evaluative activities corresponding to the sequenced decision-making 
areas as follows: 



1, S)'stems assessment 

2 , Planning 

3, Program implementation 

4, Program improvement 

5, Program certification!/ 

Thus the evaluator collects different kinds of information 
depending on which stage of decision -making is involved. The evalua- 
tion systems framework, described previously, is used primarily in 
program evaluation. To a certain extent then, those definitions of 
evaluation which focus on input -output relationships, tend to ingore 
these other important decision-making steps for which evaluative 
exexpertise may be needed. The CIPP and CSE models are important in 
that they call attention to these other steps. However, their limitation 
is that they are based on an ideal model of dec is ion -making; they 
assume that the decision maker can and actually does conduct a need 
assessment and proceeds rationally to allocate resources to programs. 
The reality is that often such is not the case; the evaluator is 
called in after the decision has been made to initiate a program, and 
often that decision has not been based on a needs assessment or on 
careful planning. Given this less than ideal situation, however, it 
may still be possible to evaluate a program. That evaluation is ac- 
complished by describing a program in terms of its inputs, outputs 
and processes--as in the first definition above. 

In summary, note the two iirportant characteristics of the defi- 
nition we have been discussing. First, it introduces the decision- 
maker's subjectivity into the analyses at the same time that it removes 
the evaluators' subjectivity. Second, it implies that the evaluator 
should take part in each stage of dec is ion -making, although realis- 
tically this is not always possible. Thus, there are some situations 
for which this definition of evaluation is clearly appropriate, and 
others for which it is not. 

Evaluation as the assessment of worth- -To many the term "evalua- 
tion" has always ijnplied the assessment of worth. Reflecting this 
viewpoint, Suchman states, "An20valuation is basically a judgment of 
wortji-- an appraisal of value."—' 

1/ Ibid,, p. 3, 

^ Suchman, op. cit., p, 11, 
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Any such judgment or appraisal is necessarily based on value judgments > 
This poj-nt is the basis for one of the controversies today in the 
field of educational evaliiation. There are some who maintain that value 
judgr/ients JvsAst be placed outside of the domain of the evalua tor --that 
the evalui^^tor merely provides information to someone else (usually 
i,the educational administrator) who then may make value judgments. Others, 
notably Scriven, Glass, Stake, and Suchman, hold that the evaluator 
mi:ist somehow be responsive to this question of values. The nature of 
that responsiveness might take the form of merely identifying and 
making explicit those areas where values enter an analysis, or of 
actually imposing the evaluator 's own values onto the situation. Glass 
for example takes the second position: 

Being of assistance to the program personnel --so 
they may better conduct their business --is a pro- 
ximate aim of evaluation; the ultimate aim of an 
evaluation is to decide questions of worth . An 
evaluator's rendering of judgment on the composite 
value of an educational program poses a threat to 
teachers and administrators, whom he might live with 
more amicably "in a service capacity." Nevertheless, 
he is obliged to make the judgment; he cannot safely 
shirk the obligation, 

1/ ^ ^ 



Glass was reacting against those who would leave the question 
of values to the decision maker- -usually to the eudcational adminis- 
trator. He argues that by so doing, important political and economic 
issues or values may be overlooked. While he may be right, this does 
not seem to justify going to the extreme of replacing the decision 
makers' subjectivity with that of the evaluator. Rather, it would 
seem that the evaluator should confront the issue of values by 
identifying and making explicit those areas where A/alues are likely 
to enter into the evaluative process. Suchman states this position 
quite convincingly: 

The process of evaluating is highly complex and sub- 
jective. Inherently it involves a combination of 
basic a.ssumptions underlying the activity being 
evaluated and of personal values on the part of both 
those whose activities are being evaluated and those 
who are doing the evaluation. Evaluation is a con- 
tinix)us social process, rarely stopping to challenge 
these assumptions or to bring the values into the 
open. The task for the development of evauative 
research as a "scientific" process is to "control" 
this intrinsic subjectivity, since it cannot be 
eliminated. 2/ 

17 Gene V. Glass, "The Growth of Evaluation Methodology," Evaluation 
Workshop of the Adult Education Research Conference, Minneapolis 
March, 1970, p. 29. . ^ » 

2/ Suchman, op. c: '.t., p, 11. 



ERLC 



-44- 



Notice that he distinguishes between evaluation and evaluative 
research, the first being the judgment of worth and the second being 
the use of scientific methods in the service of evaluation. Since the 
distinction is not one common in the lietrature of educational eval- 
uation, we prefer not to confuse the issue by using Suchmian's defi- 
nitions. Nevertheless, his treatment of the subject is quite out- 
standing and presents a strong case for training evaluative personnel 
to be aware of and to be able to deal with questions of value. 

We suggest that evalioators should in fact be trained to 
identify and jnake explicit those areas where values enter analysis. 
Beyond tliat the question of "whose values" may carry him into issues 
of accountability and community control with which he may or may not 
wis]i to become involved* Since, as stated above, we are particularly 
concerned with training administrator-evaluators, we v>rould expect 
that some of our students would in fact wish to pursue these issues. 

Conculsion — In this section we have tried to show tliat there are 
several different viewpoints about what constitutes evalioation, and 
thcit each may be appropriate to some situations and not to others. 
We feel that a training prograjn should make the student aware of 
each of thse viewpoints, and indeed the program outlined below touches 
briefly on each. 



3. Orientation to Fieldwork and Qualitative Methodologies 

The attempt to evaluate educational programs is only a recent 
concern; there is very little scientific knowledge about educational 
programs, on which to base current evaluations. For this reason we 
believe that it is important for evaluations to be heavily field oriented, 
employing qualitative -as well as quantitative methodologies. Field 
research is required in order to isolate key process variables or to 
search out anticipated variables. Because the educational process 
itself is complex and so little understood, it is all the more 
critical that field methodologies be effectively employed to sort 
out the complex that can be understood only by observing it. 

Our o\m approach to evaluation is based on three assumptions 
related to this issue of field research: 

1. E^^aluation of educational programs is a complex 
task which cannot be lijnited to the measurement 
and reporting of essentially quantitative outcomes 
for experimental and control groups using the 
pre-and post- testing model. 

2. The evaluation of operational programs in edu- 
cational settings requires a field rather than 
laboratory orientation which must seek its metho- 
dology and sociology as well as psychology and 
refine these methods in an educational context. 
Moreover, educators and educational consumers 
supervisors, teachers, pupils, and parents --must 
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be involved with educational specialists for, 
in our view, the evaluation of educational programs 
is in many ways a unique process \^^Lich must grew 
out of a fundamental understanding of education 
as well as competence in research. 

3. Finally, we proceed from the assumption that while 
educational programs of the school and the com- 
munity operate in the real world and thus cannot 
be manipulated in order to satisfy the canons of 
controlled experimentation, it is possible to be 
rigorous and empirical in field oriented research 
and evaluation- -if the objectives of the program 
being evaluated are clearly stated and if the 
methods for the collection of qualitative data 
are subjected to the same scrutiny and standar- 
dization as methods used. in quantitative analysis. 



Against this backdrop of perspectives, we ourselves are now 
pursuing attempts to assess the nature of selected educational pro- 
grams. These attempts are aimed at an examination of the content 
and process of entire programs from planning to implementation, and 
the resiiting impact on performance in school community. Research 
is based primarily on a method which v/e call Situational Analysis, 
whe^^eby on-site observers record and evaluate what happens --in 
this case, what happens in the program and in the schools and com- 
imjnity where they work. The observers bring to their field work some 
questions or predefined issues, but these preconceptions are con- 
stantly redefined in light of their obsei-vations, and new information 
gleaned from written reports of othe comparable programs and visits 
to the sites by the resident staff. The process of redefinition 
occurs at periodic work sessions involving all field observers, 
resident research staff, and consultants. These meetings serve not 
only to generate ideas rooted in actual experience but to guarantee 
standardization of the issues (refined or revised) which mold the 
field workers' observations. From the system itself, then, majoi 
issues emerge as the criteria on which final evaluation is based. 

attempt to provide assistance to the programs under study 
by feeding back data gathered by field observers. In this way, 
the impetus for change is embodied not only in the final reports but 
in the research method itself. As a process extending over time, 
we feel that this type of evaluation has greater potential for in- 
fluencing administrators than the more traditional evaluative report 
issued as a final dictum, an outside judgment capable of generating 
as much fear and resentment as change and growth. 

The feedback process also contributes to the research method 
itself. By providing program administrators with information gathered 
in the field, we can elicit their reactions to th3 research and thereby 
sharpen the relevance of issues and techniques as the study proceeds. 
IrreleA^nt questions can be discarded while new ones may be articulated. 
The reactions of administrators thus help to redefine and refine the 
basic issues of the analysis, 
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b. The Trainiite Model 

The training model proposed hire develops out of a coinprehensive 
survey of materials on evaluation ana qualitative research, and out 
of our own recent e^q^erience with several large-scale evaluations. 
The model has three conponents ; 

1. Exposure to a comprehensive overview of issues rele- 
vant to evaluation and fieldwork. This e^q)osure 

' is provided by two introductory "core" courses--, 
one on issues broadly relevant to the task of 
evaluation, and one on fieldwork and qualitative 
analysis. Course outlines are presented in Section 
1 below with proposed readings and exercises 
given in the appendices. 

2. Practical experience as a "junior" team member in 
carrying out an evaluation- raccompanied by a 
seminar to facilitate conceptualization and pro- 
vide feedback. Our own experience, and suggestions 
deriving from that experience, are presented in 
Section B. 

3. In-depth training in one or several specialty 
areas. Some thoughts about this component are 
presented in Section C. 
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1, A PROPOSAL FOR TWO COURSES 



COURSE ONE; An Overview of the Iheory of Evaluation and the Ratio- 
nale and Design of Evalijative Studies , — The outline for this course 
enconpasses a wide variety of issues pertinent to evaluation in educa- 
tion. Each issue is presented in a lesson whidhi is acconpanied by 1) 
Comments for the instructor on the issue and its relevance to evaluation, 
2) An introductory reading for the student, 3) Where appropriate, sug- 
gested exercises to provide the student with some experience in dealing 
with the particular issue and 4) A list of suggested readings for the 
instructor and the student. 

Recall that a fundamental conponent of our training model is the 
internship as a member of a team-evaluation. Before joining that team, 
interns need an overview of what evaluation entails and how their speci- 
fic "piece" of the evaluation integrates with others. This course is 
intended to provide the overview. It is an introduction. It will not 
produce skilled evaluators, although it can serve to introduce a pro- 
gram of formal training which will produce such evaluators. 

Recall also that we previously discussed several sets of viewpoints 
about what evaluation entails , and we said that it was not our intent 
to advocate one over the others. Rather, we feel that students should 
be aware of several viewpoints and should be able to adapt to particular 
situations. For this reason, in the materials which follow, we do not 
atteii5)t to "teach" any one model-such as the CIPP, Provus, or CSE Models; 
rather we suggest to the instructor where and how these models relate 
to the issues at hand. The instructor is free to pursue any model ac- 
cording to his preference. Because the course is to serve as an intro- 
duction to a particular local evaluation, we suggest that the instructor 
use materials and exanples from that situation, as the basis for many of 
the exercises suggested in the course lessons • 

Thus we have designed the course in such a way as to introduce the 
student to in5)ortant issues and to encourage the instructor to adapt the 
lessons to the particular situation of the local evaluation. For these 
reasons the materials are not tightly programmed and much of the burden 
for success of the program must rest neces:sarily with the instructor. 

Course One consists of six separate lessons, with purposes as fol- 
low: 

• Lesson One is intended to acquaint the student with the term 
"evaluation" and with tlie different uses of that term. 

. Lesson Two is concerned with the concepts "inputs", "processes", 
and "outputs" and with interrelationships among these. 

, Lesson Three introduces the student to the system concept and 
shows how educational prograins may be described as systems. 
Uses of the system concept in program evaluation and improve- 
ment are discussed. 
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. Lesson Four is designed to acqioaint tJie student with some of 
the practical realities o£ trying to apply syster^ concepts 
to actual programs in education. 

Lesson Five focus on the application of systems analysis and 
related evaluation techniques to the decision process in 
education. 

. Lesson Six presents a discussion of a nuniber of controversial 
issues related to the evaluation field. 

Detailed on Course One is given in Appendix E. 

COURSE TW); Qualitiative Data its Collection and Analysis: An 
Approach to Problem Solving. ~ The necessity for training students to 
study and observe in natural, as opposed to laboratory, situations arises 
from the need to cope with the complexities o£ real- life educational and 
c'^cial programs. Obtaining greater understanding, and ultimately control, 
over these multi-dimensional situations requires the abandonment of a 
single factor theory --a unitary independent -dependent variable frame- 
work -- and the development of a process -analysis approach to the exam- 
ination of these kinds of situations. Accordingly we have sketched out 
a one to one and one -half year program designed to take the student 
through a series of field experiences, seminars, and related readings 
that will acquaint him with both the con^lexities and the potential rich- 
ness of the collection, analysis and reporting of qualitative (generally 
non-statistical) data. 

It should be stated at the outset that this is an introductory ex- 
posure and is not designed to produce highly skilled techiiicians . It is 
assumed, however, that the student comes with a fundamental grasp of ba- 
sic experimental design and an understanding of such related issues as 
sanpling, reliability, validity, hypothesis generation, and data analysis 
and reporting, and that he will therefore be able to make comparisons 
between his newly developing qualitative approach and the more traditional 
quantitative methods. This base of knowledge will provide an anchor to 
the student, yielding greater understanding as he moves through the 
field experiences and the readings presented. 

Recall that in Part I we suggested that the core experiences of this 
program were an appropriate introduction to a variety of careers in 
evaluation and administration. For those who would specialize in the 
analysis and the collection of qualitative data, the importance of this 
course is obvious. For those who would branch into other areas of spe- 
cialization, the course is useful in providing some basic conceptual and 
analytic tools, as well as an appreciation for the rationale underlying 
process -oriented evaluation. And specifically in the case of those in- 
dividuals who may subsequently administer or direct teach evaluations, 
this course should enhance their ability to: 1) conceptualize, initiate 
or sponsor process -oriented evaluation programs; 2) hire, train, super- 
vise, and monitor specialists in this area; 3) collaborate with these 
specialists in interpreting the findings by asking critical questions; 
and 4) develop tactics based on the findings which will suggest ways to 
manipulate program elements in order to approach program goals. In this 
way the administrator- evaluator should begin to incorporate conceptual 
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tools and experiences which will allow him to move increasingly toward 
self-evaluation o£ the programs for which he is administratively respon- 
sible. 

The course will be structured around a series of readings, a set of 
structured field experiences, and an "issues outline/' The course is 
based on the following rationale: students learn in large measure by 
doing. Questions then arise relating to what they are doing, why they 
are doing "it, and the potential inplications of what they are doing. 
These .questions are predicated on a bit of trial and error. However, 
time is precious both for the instructor und for the student. Conse- 
quently, we attenpt to short-cut the process of leaming-by-doing through 
utilizing selected anticipatory readings. For exair^le, we provide read- 
ings which relate to general overviews of what field studies generally 
do and how they are often conducted. We present materials to enable the 
student to consider the possibilities of integrating the skills developed 
in this course with quasi -experimental designs for research,!/ and of 
using qualitative and quantitative data in cor^lementary and si^^plemen- 
tary fashion. 2/ Reaidings are given which can introduce students to areas 
that are rapidly becoming specialties of those utilizing qualitative 
analysis for descriijtion and understanding -- e.g. observation of school 
systems, classroom observation, and ecological psychology. 

Our approach to selecting experiences and readings for this course 
has been eclectic. We have drawn from the fields of education, psycholo 
gy, art, ecology, sociology, anthropology, economics, philosophy and 
educational anthropology. We are especially heaArily indebted to the re- 
cent integration of the participant- observation literature by McCall and 
Simmons. 37 We have made use of both the literature, and of the insights 
of persons in these fields who have been Interested in the process of 
learning to work creatively with qualitiative data. What we sought from 
all these resources, both human and litarary, were answers (or approaches 
to answers) to the following questions: 

1) Miat is the process of observation all about? 

2) How does one learn to be a skilled observer? 

3) What conceptual steps (ox experiences) will enable a student 
to recognize his own biases aiid value judgements and how these 
may affect the "obj activity,*' replicability, and ultimately 
the validity of his data? 

1/ uonaid r. L;aJT5)beil and Julian C. Stanley, Experimental and 
Quasi -^Exgerimental Designs for Research, New York , Rand McNally and Co . , 
Reprinted irom Handbook"of Research on Teaching, American Educational 
Research Association, 1963, pp. 34-71. 

2/ Sjmuel Sieber, "The Integration of Field Work and Survey Methods," 
Bureau ot Applied Social Research, Columbia University, 1969, Prepared 
for a reading in the Sociology of Education , The Free Press (forthcoming) 

3/ George J. McCall and J. L. Simmons, eds.. Issues in Participant 
Observation, A Text and Reader, Reading, Mass., Addison Wesley, 1969. 
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4) How does one teach the student to deal with the inevitable 
interplay of theory, description and bias? 

5) How does one learn to utilize the information or hundies gained 
through this process in order to a] obtain greater understand- 
ing of his particular problem, b) discover that he may have a 
problem is already being solved by a process he can now describe 
and consider replicating elsewhere, or d) generate new questions 
which must be asked if understanding is ever to be reached? 

In sum, the basic issues of this course relate to learning to ob- 
serve, and learning how learning to observe relates to learning to ask 
questions. The generation of en^irically based theory is another possi- 
ble outcome. 

In order to give an unknown instructor a framework within which to 
integrate field experiences, i^eadings, and his own preferences as to what 
is critical, we are providing three basic documents for his consideration. 

Section I: Field Experiences. This consists of a tenporally ordered 
(an deliverately so) set of field experiences for the student which will 
require supervision, guidance and integration of the readings. As we 
have stated previously, we recommend that these field experiences should 
be the core for the course, and we recognize that we have by no means 
exhausted the potentially relevant field experiences with tJiese initial 
suggestions. 

Section II: 'es Outline. This consists of the outline of the 

fundamental issues believe should be touched upon in teaching the pro- 
cess-strategy to wl A we have alluded, have tried to organize the 
issues so that they may be easily related to the field exercises. Thus 
as students carry out the exercises outside class, the instructor may 
wish to devote class time to discussion of issues suggested by the out- 
line. Readings which relate to the issues are contained in Section III. 

Section III: Bibliography. Readings related to the issues raised 

by the field experiences and the conduct of a field study and intended 

to illustrate or elaborate these issues are provided. Particularly 
useful readings are starred. 

We have tried no plan for an experience lasting a year to a year-and- 
one-half. The length of time required would of course depend on how 
frequently the course met is and the depth in which issues are pursued. 

Detailed infonnation for the three sections of Course Two is provided 
in Appendix F. 
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2. THE PRACTICAL EXPERIENCE AS A "JUNIOR" TEAM MBvlBER 



As explained above, we velieve the practical experience with an on- 
going evaliiation to be a critical coii^onent of our training model. This 
experience exposes interns to schools and school personnel thus making 
them aware of a host of features of the "real world" that is never taught 
well in the abstract. We believe that the practical field experience is 
at the minimum as important as the conceptual knowledge provided through 
course -work. 

A second important aspect of this experience is learning to function 
as a member of an evaluation team. We are fairly well convinced that 
the team approach to evaluation is the only way to attack most issues of 
evaluation today. Again the actual experience of serving on a team - of 
learning to coordinate one's own specialty with those of other team mem- 
bers - would seem to be valuable preparation for later professional par- 
ticipation as a full team member. 

Our own experience indicates that it is possible to successfully em- 
ploy interns as "junior" team members, at the same time that they are 
pursuing related academic course work. Experience also indicates that 
interns do in fact learn a great deal about practical issues of evalua- 
tion, about sdiool systems, and about functioning ^as a team, through the 
internship. Because we had not established a degree program in. evalua- 
tion, it was necessary for-us^o find ways to combine special training 
in evaluation with other degree programs in the College; programs such 
as curriculum development and administration were particularly appropriate 
in this regard, perhaps because they too focus on the practical field 
experience. 

We brougjit both beginning and-advianced students onto the evaluation 
teajn; their background and experiences were varied. Interns were given 
an overview to issues of evaluation and then participated throughout the 
year in a seminar intended to probe further into such issues as they 
arose. Each intern was responsible for spending several days a week ob- 
serving and interviewing in one of the schools in our study, carrying 
out appropriate aspects of the study throughout the year. Each was re- 
sponsivle for participating in the work being done for the project in one 
of the following substantive areas: 

Psychcrne tries 

Community- School Relati onships 
Political Science 
Organizational Analysis 
Sociology and Survey Methodology 
Classroom Observation 
Computer Technology 
Coverage of UFT Affairs 

Faculty associates were responsible for supervising several interns 
and partici^pating in the study themselves. In most cases, a faculty 
associate supervised the study in 3-4 schools and was responsible for 
directing activities within one of the above substantive areas. A few 



of the advanced students were also responsible for directing activities 
wi thing substantive areas. 

Besides participating in an evaluation seminar which was integrated 
with their field experiences (and for vdiich intenis received course cre- 
dit) , interns also took courses in pursuit of their academic interests. 
They were encouraged to pursue courses relevant to their own work in 
evaluation - most were pursuing a specialty area in depth. Thus interns 
participated in common experiences in evaluation, and yet were enrolled 
in a variety of degree programs. 

This was an extremely practical arrangement for our own purposes in 
that it all(Med us to provide interns with the stability of established 
degree programs, while we experimented with - or field tested - some of 
our own notions about what an established degree program in evaluation 
should encompass. The arrangement was made possible financially by the 
fact that we were funded for a large-scale evaluation for which student 
field workers would receive a modest fee for fieldwork and other pro- 
fessional contributions to the project. That fee was easily conpetitive 
with stipends offered for many other graduate programs. Our interns were 
required to perform work for that fee, but by integrating the work ex- 
perience into course work (i.e., by granting some course credity) and by - 
relating that experience to professional interests and growth, it seems 
to us that this particular work- study experience can be an attractive 
package for students. 

We also built faculty associates into our staffing pattern and con- 
tracted specifically for released time from course work so that faculty 
could participate in the study and provide interns with intensive super- 
vision and guidance. This assures the grantor that excellent professional 
supervision will be provided. Use of interns who receive a modest, but 
conpetitive reimbursement also makes it possible to provide extensive 
field coverage at a manageable cost. 

Summary . Based on our own experience, we suggest that the fol- 

lowing program for interns as '^junior" team menibers is feasible and 
desirable: 

1) Interns participate as members of a team evaluation group. 
It is possible to. employ interns with a range of expertise. 
The type of participation which is appropriate depends on 
the students' own backgrounds, experiences and expertise. 

Generally, advanced students can be assigned positions of 
responsibility while beginning students can be prepared to 
move into such positions. 

2) Since there are very few training programs in evaluation per se , . I 
it can be anticipated that few students, not even advanced 

students - will have been exposed systematically to issues of 
evaluation or of field work. We suggest that the introductory 
training program described above serves as an appropriate 
introduction. The introductiiDn to issues of evaluation might 
be given even before foimal participation in the project. 
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The introduction to field work would probably have to be pro- 
vided throughout the year. 



3) The internship experience should be accon^anied by a continuing 
seminar which addresses the theoretical and methodological is- 
sues that arise in the field evaluation project. 

- 3 Interns should pursue course work and a degree in some area of 
specialization. Many areas are appropriate. 

5) Faculty should be freed from some of their teacliing responsi- 
bilities in order to direct inportant substantive areas of 
the evaluation project and to provide supeivision and guidance 
for interns. 

6) A work- training package such as the one described above can be 
made attractive to interns who seek stipends and to contractors 
who desire maximum coverage at minimum cost. 

7) By utilizing existing degree programs at the same time that 
common^experiences are provided for all interns, it is possi- 
ble to build a training, program in evaluation into existing 
institutional arrangements. When definitive plans for a de- 
gree program in evaluation are established, it is possible then 
to move toward the formal establishment of such a program. 
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3. IN-DEPTH TRAINING IN A SPECIALTY AREA 



Each intern should pursue in-depth training in one or several special- 
ty areas related to evalioation. These areas might include, but are not 
limited to the following: 

Administration 

Curriculum Development and Evaluation 
Psychome tries 
Survey Research 

"Foimdations^' such as coicology, economics, political science 
Classroom Observation 
Organizational Analysis 

Fieldwork and Methods o£ Qualitative Analysis 

Since currently most programs in these areas are not oriented specifical- 
ly toward issues and techniques of evaluation, some thought needs to be 
devoted to the nature of the in-depth experience for each area. The 
intern and his faculty advisor might do this on an ad hoc basis for be- 
ginning programs, but as the programs develop at an institution, a 
faculty program committee should assume this function. 

In this section we provide an example of one in-depth experience in 
a specialty area oriented to issues and techniques of evaluation. We 
selected the area of "Classroom observation" for these purposes. This 
seemed inportant first because of the obvious importance of classroom 
observation in conducting evaluative studies. Second, there seems to 
have been little thought to what comprehensive training in this area 
should entail; our proposal is a preliminary effort to provide some 
thoughts about this question. An intensive year-long course is proposed 
(the equivalent of four one-semester courses) . A bibliography for the 
instructor and students is proved in Appendix G. 

a. Systematic Observation of the Instructional Process - Proposal 
for a Year Long Course 

Introduction. -- In order to prepare interns with an expertise in 
the area of classroom research, the following year long course is pro- 
posed. The course has been outlined under the assumption that students 
entering it will have already had an introduction to qualitative methods 
and observation. Additionally, to qualify them as specialists in the 
area of classroom research, during the course of their studies in a grad- 
uate program preparing them in evaluative research, they should also be 
exposed to data analysis techniques, basic statistical techniques, design, 
and sampling procedures, and a survey course in developmental psychology. 
This last is particularly important for persons interested in the early 
elementary years. The Instructor is assuimed to have had experience in 
conducting classroom observations and to have a loiowledge of the litera- 
ture in the field. 

The proposed course will serve as a thorough introduction to the 
field of classroom observation. It will emphasize theoretical issues in 
the area of classroom research and their implication for an evaluation 
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studjr. It will provide students with m acquaintance with existing 
instruments and good knowledge of one ox two. Central to the course will 
be a planned laboratory experience which will involve regular observation 
in functioning classrooms throughout the year. The laboratory sessions 
will be coordinated with the lecture discussion sessions, A3 outlined 
below the course would require at least double the aitount of work in and 
out of class, as compared with most graduate courses, and should be 
credit accordingly. 

Lecture discussion sessions, In the lecture-discussion sessions, 

the instructor should introduce the students to the literature in the 
field, by first discussing in detail actual published reports of research 
losing representatives forms of classroom observation instruments. The 
reports to be read and discussed should include qualitative and non- in- 
strumented work such as Jackson's and that of Smith and Geoffrey, ecolog- 
ical frameivorks, such as Gump's, sign systems, rating scales and reports 
presenting analyses of video- and/or audio- tapes. The purpose in read- 
ing and comparing these reports would not only be to acquaint the stu- 
dents with some systems (not to teach them to use the system), but more 
importantly to illustrate the various t/.-^ortical issues in the field of 
classroom research, Distussion should cover the following topics: Pur- 
pose of observational research, research strategies, relation of obser- 
vational research to courses of instruction and curriculum, conceptual 
frameworks, observer stance, empirically versus prescriptively oriented 
studies, techniques of data collection, types of instruments and associa- 
ted strengths and weaknesses, problems of reliability, validity, levels 
of inference, particular dimensions and concepts typically applied to 
classrooms, treatment of variables, kinds of classrooms sampled and 
effects on the instrument, age level of pupils in relation to the appro- 
priateness of the behavior collected. Appropriate readings and review 
studies should supplement research reports for the students. 

The second half of the course should concentrate on the function of 
observation of instruction in an evaluation study. Again the class should 
read and discuss some evaluation studies in which classroom observations 
fromed a part of the analysis. From this reading should come an under- 
standing of the contributions that ovservations of functioning class- 
rooms can make to new or ongoing instructional programs, new curricula, 
new administrative arrangements, or new educational objectives. Dis- 
cussion of the use of observational dr.ta should be in terms of extent of 
implementation, possibilities for feedback to the program implementers 
and to teachers, and for evaluation of the success of various objectives. 
Classroom data should be considered, thus, in terms of some of the pro- 
posed models (Stake, Provus) for doing evaluative research. Time should 
also be devoted to contrasting studies to be done from an evaluative 
framework with those proceeding from a basic research point of view. 
Classroom researdi should be compared to observational studies in other 
fields such as study of small groups or others , depending on the back- 
ground and interests of the students and instructor. Finally, there 
should be some time devoted to reading and discussing possible new sub- 
stansive sources for concepts to be used in devising classroom observa- 
tions and ways of developing instruments from such concepts, for example 
decision theory or linguistic analysis. The sources chosen would depend 
on the strengtli of the instructor and students. 
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Laboratory sessions. A primary requisite for the planned lab- 

oratory session would be to make available to the students convenient 
functioning classrooms at a range of age levels from early primary to 
at least high school (perhaps same instructors in the graduate school 
would pemit their own classes to be observed) and preferably, some range 
of educational philosophy represented. It would advisable for these 
classes to be observed by the instructor to check if different educational 
strategies are indeed being practiced prior to sending the graduate stu- 
dents to the classes. The students should not have to search for the 
classes, they shouM be provided vith them and the course requirements 
should mandate regular observations in tliese classes by the students en- 
rolled. Video and autio-tapes could supplement but not replace such 
live observation. 

At the beginning of the year, students should enter and observe the 
classrooms in a relatively untutored fashion, and then after one or tvvo 
such observations, and written presentation and subsequent discussion of 
vihat was found, sutdents should be sent in groups to the classes with 
instructions to reproduce a set of Jackson, Smith and Geoffrey, Gump, and 
Barker and Wright type observations. Possibly four persons each assigned 
to collect one kind of observational ^-ecord would be assigned to the same 
classroom. This wouI.d produce matej*ial for di.s cuss ion and comparison. 
An important feature of the laboratory sessions is the written presenta- 
tion and discussion of the observations made at all stages during the 
year. If such systematic reporting is not done the experiences of the 
students will not reach the desired level of organization and sophistica- 
tion, but will remain a concrete and unsystematic collection of impressions. 

After this introductory period the students should divide themselves 
into taks groups of three or so, each of which would concentrate on a 
set of related concepts (such as cognitive, or ecological or affective 
or linguistic, etc.) and perhaps depending on the number of groups, a 
certain age level. The groups should form to capitalize on the students* 
own areas of substantive strengths or interests. As issues and topics 
arise in the lecture -discuss ion sessions, each group of students would, 
with the supervision of the instructor select existing instruments or 
parts of instruments and use them to make observation and/ or to analyze 
recorded class sessions. For example,, a group interested in cognitive 
concepts would use various types of instruments that focus on cognitive 
variables, such as sign instruments and rating scales, as well as try out 
category systems on audio recordings. Similarly various kinds of units 
would be exemplified by the practice observing, and so would systems 
that require varying degrees of inference, in addition to gaining ex- 
perience with various kinds of instruments, the analysis of the data 
would allow experience with problems such as observer reliability and 
validity, and permit evaluation of the appropriateness of various systems 
and variables chosen for the behavior of the pupils and teachers being' 
observed. 

The instruments could be selected from the eighty or so systems re- 
produced in Mirrors For Behavior as well as from any other convenient 
source. The presentations and discussions should include comparisons 
among the various instruments as well as evaluation of how well the sys- 
tems handle the substantive concepts involved. 
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Thus the first semester should be devoted to using existing instru- 
ments in a schedule coordinated with the lecture-discussions. The aims 
of th.e laboratory experience would be to give practical experience in 
using the instruments. The final task of the first semester should be 
to collect another set of f ieldnotes v^dch would be compared to those 
collected during the first weeks of class. It will be interesting to 
observe the changes the students perceptions of classes will have under- 
gone as a result of their structured obseivation experiences. 

The second semester should consist of two basic tasks. First the 
development of observational systems as part of various hypothetical 
evaluation programs, and secondly, the designing of an observational sys- 
tem in line with the student's interest, or alternatively, the refining 
of the instrument set up in the hypothetical evaluation program. 

The instructor would provide a series of typical programs that might 
be evaluated. The prograjns could be curriculum programs, new teaching 
arrangements, new instructional techniques, and so forth. From these 
candidates, the students, in groups, would suggest a classroom observa- 
tion instrument or analysis program ^ either by selecting from among 
appropriate existing instruments, or parts of instruments, modifying 
existing instruments, or by designing original instruments. The students 
would be required to outline the function of the instrument in terms of 
the evaluation program, perhaps fitting it into some particular model 
of evaluative research. They would have to justify its use both by the 
kinds of data it would provide to the hypothetical clients and in terms 
of what il: souie contribute to the field of classroom research. This 
project could be combined with the second task of tlie course by having 
the student pursue his om research interest with the instrument, for 
example by inserting some research items, etc. 

In any case, sample observations should be collected and analyzed 
and a protot>T)e report be written and presented. Just as it was impor- 
tant during the first half of the course that written presentations of 
the observations be prepared and analyzed in systematic form, so it is 
essential in this part of the course that reports be prepared for the 
various stages in an observational study. For example, a proposal 
should be compiled for the justification of the particular instrument 
selected and the paper should be written and presented as if it indeed 
were an application for performing an observational study in the hypo- 
thetical evaluation program. The reports would be discussed with the 
instructor and compared with reports from other groups for differences 
in orientation and instruments due to the different circumstances of 
the hypotetical evaluation programs. 

This work this second semester vA.'iuld allow the student more in- 
dependence and freedom to pursue his own particular interests. Addition- 
ally it would provide the student with practical experience in instru- 
ment development and instrumentation. In fact, having a range of class- 
rooms available for observation purposes would allow the students to 
evaluate the effect of various kinds of classrooiTiS on the development 
of instruments. It might even be useful to have a ^specialist in instru- 
mentation available for consultation by the students. Finally the ex- 
periences in the field, in conjunction with the lecture discussion 
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sessions would force the student to translate his theoretical knowledge 
and conceptual experience into stratjegies for analyzing real behavior. 
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Y. CONCLUSIONS AND RECOMMENDATIONS 



Overall, the project can be counted a success. It generated and 
successful tested Jimdels for solving some of the most difficult school 
evaluation problems c Within this larger context, it identified a number 
of procedural problems and suggested ways these should be overcome. 
Finally, it provided a success experience for about 80 percent of the 
persons vdio entered it either as trainees or interns. 

Since conclusions and recanmendations are spelled out in the 
descriptions of individual ccmponents on preceding pages, they will not 
be treated in detail here. A few summary stataments are in order, however. 

The portion of the project carried out by the Council of the Great 
City Schools provides a model for upgrading the skills and greatly in- 
creasing the perspective of experienced school research and evaluation 
specialists. This project component also gave the Council a greatly 
increased research capability and allowed it to focus on a number of 
research -related urban school problems in. a way which would otherwise 
have been impossible. The Council derived a great deal fron the project 
and naturally would like to see it refunded. Within this context, the 
primary recommendations have to do with recruiting. In addition to 
experience in conducting research and evaluation activities in the 
school setting and an understanding of urban education and its problems, 
interns for this type of program must have an area of special interest 
theywishj to pursue plus the ability to work independently and take 
responsibility without close supervision. Recruiting and personnel 
assessment procedures must be carried out so that the requisite skills 
and abilities are known to be present. 

The coii5)onent managed by the School District of Philadelphia pro- 
vides a workable method of training persons especially for research and 
evaluation work in urban school districts. Recruitment of entry- level 
personnel in this field has been a serious problem in the large city 
school systems, and the Philadelphia experience illustrates a way this 
probl^ can be overcome. A number of recanmendations for ijnprovement 
of the operation of the model were listed previously. In general, the 
recommendation would be to institute similar training models in a number 
of locations throughout the country* The city school districts to be 
involved would need to havo a top-quality research and evaluation 
operations, and there \/ould need to be a local university, willing to 
cooperate and with quality course offerings in the technical areas. 
The Office of Education might set up a special subsection in its 
Research Training Branch for urban education, and a number of projects 
in different locations could be funded. 
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Teachers College, Columbia University developed a new evaluation 
model in its part o£ the project. Some trial of the model has been 
undertaken and it appears premising. Development is in the early 
stages, however, and a good deal o£ additional trial and assessment, 
revision, and retrial is needed. The model holds great premise for 
school districts because it is based in the reality \diich they face. 
Development needs to continue. The model should be tried in a number 
of school districts. Plans for training in. use of the model need to be 
completed and a training program instituted, jend persons so trained 
should be placed in school research" divis'ions and their experiences 
assessed. In short, develqpment of the model needs to be continued 
and then the model implemented. Only when the job is completed can 
the work at Columbia be of practical value to operating school systems. 
Further funding of the Teadiers College work in this area is reconmended. 

Obviously, the evaluation problems of the large urban school dis- 
tricts were not solved through xhe project. An important first step 
toward developing solutions to these problems was' taken, however, and 
it is not of great iJiportance to continue and complete the task. 
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Appendix A 

Introductory Section o£ the Report on Test Use in 
The Great City Schools 



TESTING PROBLmS 



Critics o£ testing have pointed out that the tests being used. are or- 
iented to middle-class values because the test instruments are "standard- 
ized'^ upon samples o£ children drawn predominately from the middle class 
milieu. It is charged that the test items reflect unfairly upon inner 
city and disadvantaged school children whose limited experiences deny 
them the opportunity to perform well on most tests currently in use. It 
is argued that tests and test items must be more ''relevant'' to children's 
environmental experiences and that measurement problems are compounded 
by different learning and life styles. 

Problems of measurement exist with children of different cultural 
heritage. Children with multi- lingual or non English-speaking back- 
grounds are placed at great disadvantage in the classroom. They lack the 
level of English fluency of their classmates. Such children tend to 
become separated from regular classroom settings for two reasons: 

1) Tests that draw heavily upon verbal skills 
(English) discriminate against these chil- 
dren, 

2) By the fact that nearly all classroom cours- 
es are taught in English, instruction in na- 
tive English enhances the language sophisti- 
cation of the average, middle-class American 
child, while the English-as -a-second language 
child is attempting to grasp enough lingual 
skills to function with the subject matter at 
hand. Tlierefore he tends to drop behind. 

The use of test results becomes an important issue. If children are 
placed in learning tracks on the basis of questiona e test scores, they 
may be placed in classes that fail to stimulate their intellectual growth* 
These children become part of a "locked- in," immobile instructional en- 
vironment that is socially isolated. The result is an ever -widening 
gap between these culturally-different students and children of middle - 
class American Heritage. 

COUNCIL INVESTIGATION 

The Board of Directors of the Council of the Great City Schools has 
directed its research staff to investigate issues surrounding testing in 
the Great City Schools and the subsequent use of test results. 

To proceed with its investigation on a sound basis, the Council re- 
search staff developed a Survey of Testing questionnaire that was mailed 
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to each testing director in the Great Cities. The purposes of this sur- 
vey were tiie follovdng: 

1. To detennine to what extent the large, 
urban school systems, particularly the 
Great Cities, have been/are/will be in- 
volved in test Instrument standardiza- 
tion. 

2. To sample several aspects of the use of 
test results. 

The Survey was divided into three parts; 

Part 1 Information concerning test standardi- 
zation 

Part 2 Information concerning the use of test 

results 

Part 3 All test currently used in the Great 

City Schools 

This Document Summarizes the returns from the Great Cities. 

GENERAL OBSERVATIONS 

Test Standardization 

Much criticism has been directed at school systems for testing dis- 
advantaged, minority-group childrein with test instruments that were 
standardized and norm.<2d on middle-class children. 

The school population of the Great Cities' Schools (approximately 
4.5 million) predominantly inner-city and disadvantaged. One objective 
of this Sur\''c?y of Testing was to gain some perspective on the extent to 
which children in our i?.ember city schools have been included in the sam- 
ples drawn by test publishers. While data gathered on test-standardiza- 
tion participation (past, present, and future) cites only a crude city- 
by-city measure, without regard for sample sizes Within those cities, 
the data indicates that, thus far, the mos^ widely used test -- the 
Stanford Achievement Test will draw standardization sanples in only 
two of the Great Citfes in the 1973 edition. 

Use of Test Results 

Two items were included in the survey to determine how test results 
are used. Eight of eighteen Great City School Systems have had their 
test results published in local newspapers. No effort, to our knowledge, 
has been made to evaluate this public relations aspect of testing programs 
or the political repercussions upon large city school systems due to 
varying policies in releasing or withholding test results. 
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The more important itesm in this section revealed that five large 
city systems aclaiowledged placing students in learning tracks on the ba- 
sis o£ test results. Four school systems indicated the tests they use 
as one factor in student placement procedures. Three systems indicated 
the use of intelligence or mental ability tests; one system cited achieve- 
ment tests only. 

Individually-Administered Mental Ability Tests 

The Wechsler Scales and the Stanford-Binet appear to be used concur- 
rently by the same cities in their Psychological Services, ESEA, and 
Follow-Through programs. 

Group Mental Ability Tests 

The Lorge-Thomdike Intelligence Test is being used by eight of the 
Great Cities, the Kuhlmann -Anderson is used in five Great Cities. 

Readiness Tests 

The Metropolitan appears to be the most widely accepted readiness 

test. 

Achievement Tests 

The Stanford Achievement Tests are the most popular of the achieve- 
ment tests. Other widely used tests are the Cooperative Tests, the 
Metropolitan Tests, the California Tests, the Comprehensive Tests of Basic 
Skills, the STEP Tests. The Gates -MacGinitie Reading Tests appear to be 
a popular test for non-regular school programs. 



TESTS STANDARDIZED SINCE 1964 AND THE PARTICIPATING GREAT CITIES 



Academic Interest Index 



Milwaukee 



Analysis of Learning Potential 



Philadelphia 
San Francisco 



Bailey Infant Scales of Developrrient 
Standardization Pro j ect 



Detroit 



Preference Study Using Bender-Gestalt Test 
With Children (3-8) 



Detroit 



Bennet Mechanical Comprehension Test 



St. Louis 



Bilmore Oral Reading Test, Series II 



Milwaukee 



Boehm Tests of Basic Concepts 



Milwaukee 
Philadelphia 
San Francisco 
St . Louis 
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ERIC 



California Tests of Basic Skills 

California TEst of Basic Experiences 

Cognit.we Abilities Test 

(Item Analysis § Standardization) 

Contemporary Mathematics Tests 

Content Evaluation Seriei* 

Cooperative Primary Tests 

Comprehensive Tests in Basic Skills 



Detroit Reading Readiness Test 
(validation of revised test) 

Dictation Test (Seniors) 

Dimensions of Temperament 

The American Association of Health, Physical 
Education § Recreation Physical Fitness Tests 
(Special Education Classes) 

Health Education Standardization Test 
(Teachers) 

Metropolitan Achievement Tests 

Metropolitan Achievement Tests Standardization " 
(Program 2-8) 

New York State Arithmetic Test (1969) 

Otis-Lennon Item Analysis Research 

Otis-Lennon Mental Ability Test 

Otis-Lennon Mental Ability Test Normative Study (1 

Personnel Tests for Industry 

Pimsleur Language Aptitude Battery 

Pimsleur Language Aptitude Battery (validation, 
grades 9-12) 

Primary Achievement 
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Buffalo 
Milwaukee 

Memphis 



Milwaukee 

Milwaukee 

Milwaukee 

Milwaukee 

Cleveland 
Portland 
San Francisco 
St. Louis 

Detroit 
Detroit 
Milwaukee 

Detroit 

Detroit 
Memphis 

Memphis 
Buffalo 
Chicago 
Chicago 
■3) Detroit 

Philadelphia 
Milwaukee 

Detroit 
Portland 



quick Test (restandardization, K-123 
Social Studies Achievement Test 
Stanford Early School Achievement Test 
Structure of Number Systems Test 
Survey Reading Tests 
S.T.E.P. 

S.T.E.P. Series II (Norming Program, 3-12) 
SCAT 

SCAT (grades 9 ^ 10) 
SCAT (normative study - grades (10-12) 
SCAT II 

Tests of Basic Experiences 

Tests of Basic Ecnomcs 

Typing Tests for Business 

U. S. Armed Forces Institute Course Exam 

Wisconsin Contemporary Math 

TEST STANDARDIZATION INFORMATION 



Detroit 

Detroit 

San Francisco 

Detroit 

Portland 

Portland 

Memphis 

Cleveland, Detroit 

Manphis 

Detroit 

Por :land, Milwaukee 

Philadelphia 

Milwaukee 

San Francisco 

Milwaukee 

Milwaukee 



2. Is your school system currently involved in any commercial test stan- 
dardization effort? 



Yes 8 



No 10 



Current Test Standardization Efforts Involving Eight of the Great Cities: 



T(3St 

California Achievement Tests 
Cooperative Tests 

Iowa Tests of Basic Skills 

Metropolitan Achievemer-t Test 
(1970 Edition) 

Modem Economics Test 

National Educational Develop 
ment Test (1970) 
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Publisher 

McGraw-Hill 

Educational Testing 
Service 

Houghton-Mifflin 



Houghton-Mifflin 

Science Research 
Associates 
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City 

Chicago 

Baltimore 

Portland 

Chicago 
Pittsburgh 

Milwaukee 
Memphis 



Short-Form Test of Academic 
Aptitude 

STEP II (Sequential Tests o£ 
Educational Progress) 



SCAT 

Tests o£ Basic Experience 
Wisconsin Modem Math Tests 



Educational Testing 
Service 



Educational Testing 
Service 



McGraw-Hill 



Chicago 

Baltimore 

Chicago 

Detroit 

Milwaukee 

Philadelphia 

BaltiJTiore 
Detroi t 
Philadelphia 

Baltijnore 
Milwaukee 



3. Has your system been selected by a test publisher for participation 
in future test- standardization efforts? 



Yes 



13 



No 



4. Have you agreed to participate in such future efforts'! 



Yes 



No 



Under 

Consideration 



No Response 



Future Standardization Efforts in Nine Great Cities: 



Test 


Publisher 


City 


Date 


Iowa Tests of Basic 


Houghton-Mif f 1 in 


Atlanta 


Fall 197D 


Skills 


Chicago 


Fall 1970 






Cleveland 








Detroit 


Fall 1970 






(primary battery, 


Spring 1971) 






Philadelphia 


Spring 1971 


Stanford Achievement 


Harcourt, Brace ^ World 


Baltimore 


Fall 1970 


Test (1973 Ed.) 




Milwaukee 


Fall 1970 


Tests of Academic 




Chicago 


Fall 1970 


Progess 




Detroit 


Fall 1970 


Cognitive Abilities 


Houghton-Mifflin 


Philadelphia 


Fall 1970 


Test 




Detroit 








Chicago 




National Ediocational 


Science Research * 


Cleveland 




Developsfnent Tests 


Associates 


Chicago 


Fall 1970 






Minneapolis 


Fall 1970 






Washington, D.C. 


1970-1971 
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USE OF TEST RESULTS 



5. Does your school system publish test results in local newspapers? 

Yes 8 No 10 



Great Cities that have had test refults published in local newspapers: 

Baltiinore 

Qiicago 

Detroit (1969) 

District of Columbia 

Los Angeles 

Philadelphia 

San Diego 

San Francisco 

Comments : 

Results are released to newspapers for publication 

The press repeats reports made to Board o£ Education 

6. Does your school system place students in learning tracks on the 
basis of test results? 

Yes 5 No 12 No Response 1 

Co mment s : 

1) Partially 

2) "Learning Tr£.'cks," No; "Guides for Instruction," Yes. 

3) Test scores are used in conjunction with other information in 
the assignment of pupils. The concept of a "learning track'' 
is not emplo>^d in our system. 

4) Results of all city-wide tests are used in connection with 
various class level placements, but are not a single criterion 
for programming, nor are students actually "tracked" as they 
may be enrolled "in several classes of different leve? 
Teacher's recommendation is also utilized in determnation of 
appropriate class level assignmentSn 

5) Test results are used only as a partial criteria in tracking 
reading readiness classes, classes for mentally retarded, and 
scholars (academically talented). 

6) A scholastic aptitude test score is only one element of infor- 
mation for placement in special or advanced study classes. It 
is to be used in conjunction with the child's past history 



ERIC 



-66- 



of achievement, teacher observation o£ classroom performance, 
motivation, interests, and parental opinion. 

7) For specific learning deficiencies and not on a track basis. 

Tests cited (in four districts) as one factor in student placement poli- 
cies: 

District 1. - Kuhlman-Anderson 

Stanford- Bine t Intelligence Test 

District 2. - Iowa Test of Basic Skills 
Stanford Achievement 
Iowa Tests of Educational Development 

District 3 . - Kuhlman-Anderson 
Otis-Lennon 

District 4. - WISC 

Stanford- Binet 

This section of the testinf; report provides an overview of the work 
of Ivtr. Calendine and illustrates the uiportant infoniiation he produced. 
The report also includes a 91 -page listing of every commercially-produced 
test in use in each Great City district. 
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Appendix B 



THE RESEARCH COUNCIL OF THE GREAT CITY SCTOOLS 

Agenda for Februa.ry Staff Training Session 
Rivermont Holiday Inn 
Memphis,, Tennessee 
Februar/ 25, 26, 27 



Wednesday, February 25 
8:15 



BREAKFAXr Room to be Announced. 
Opening Remarks and Introduction of 
Resource Consultants 



9:15 ISSUES IN MEASURING AFFECT : 

General Introduction to the Sessi( n 
Topic 

10:45 Break 

11:00 gviALL GROUP SESSION: Begin Outlining 
Specific Problems Experienced in 
School Systems 

12:00 LUNCH 

1:00 GENERAL NATURE OF ATTITUDE FORMATION 
AND CHANGE: Emphasis on Problems 
Identified by Participants and on 
Areas of Concern in Public Education 

3:00 Break 

3:15 34ALL GRO^JP SESSION: Continue Outlining 
Problems 

5:30 COCKTAIL PARTY 

Thursday, February 26: 



9:00 



10:30 



10:45 



12:00 



SPECIFIC ATTITUDE MEASURING 
INSTRUMENTS: Focus on the 
Likert Technique 

Break 

GROUP SESSION: Practicum on 
Developing a Likert Scale 

LUNCH 



Dr. John Hayman 



Dr. David 0:rr 



Participants and Staff 



Dr. Mar/in Shaw • 

Participants and Staff 
Participants and Staff 



Dr. Everett Rogers 



Dr. Marvin Shaw and 
Participants 
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1:30 ■ GENERAL ATTITUDE MEASURING ' 
INSTRUMENTS: Focus on the 
Semantic Differential 

3:15 Break 

3:30 SMALL GROUP SESSION: Practicum 

on Developing a Seinantic Differential 
Scale 

7:00 PRESENTATIONS BY RESEARCH STAFF: 
Experiences in Developing and 
Using Affective Instruments 

Friday, February 27 : 

9:00 MEASURES OF AFFINITY: The 
Sociometric Technique 

10:45 Break 

11:00 94ALL GROUP SESSION: Determination 
of Ways the Sociometric Technique 
Might be Used in School Research 

12:00 LUNCH 

1:30 REPORT RACK FROM SMALL GROUPS; 

GENERAL SESSION AND FINAL WRAP-UP ' 



Dr. Bradley Greenberg 



Dr. Bradley Greenberg 
and Participants 

Miss Marion Kilbane 
Dr. John Teirtple 
Dr. Leo Weisbender 



Dr. Everett Rogers 



Participants and Staff 



Participants and Staff 



3:30 Session Ends 
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Appendix C 

The Research Council of the Great City Schools 

Resenrch Directors Seminar 
Vail Village Inn 
Vail, Colorado 
May 19-23, 1970 

SYSTEMS VIEWPOINT IN PROJECT DEVELOPMENT 

Wednesday, May 20, 1970 

MORNING 
First Session 

Systems Approach in Dealing with R,D,D, and E Problems Dr. Desmond Cook 
Second Session 

Basic Systems Concepts Dr. John Skalski 

AFTERNOON 
Third Session 

Systems Tools and Techniques, I Dr. G. Trzebiatowski 

Fourth Session 

Systems Tools and Techniques, II Dr. Cook and 

Dr. Trzebiatowski 

Thursday, May 21, 1970 

MORNING 
Fifth Session 

Research Director - A Systems Manager Dr. Cock 

Sixth Session 
Systems Thinking in Single Pro;}ect Planning and 

Development Dr. Skalski 

AFTERNOON 
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Eleventh Session 



Implications of Using the Systems Approach, For 

The Research Director Resource Panel 

Twelvth Session 

What Does It All Mean? 

Where Do We Go From Here? Directors Panel 
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Appendix D 



Report of Training Institute for 
Research and Evaluation Personnel 
of the Great City Schools 



Introduction and Background 

During the dates of September 21-23, 1970 ^ Teaching Research con- 
ducted a training institute, in Monmouth, Oregon, for research and eval- 
uation personnel of the twenty-one cooperating School Districts of The 
Council of the Great City Schools. 

Preliminary planning and negotiations for the training institute 
had been negotiated with Dr. John L. Hayman, Jr. , Director of Research 
of The 'Council of the Great City Schools during the spring of 1970- It 
was agreed that a three day institute would be held during late September, 
1970, in Oregon for research and evaluation personnel of the Great City 
Schools Research Departments to be organized around the foui* areas of 
proposal writing, instructional systems, measurement and evaluation. 

It was further agreed that the following details would be included 
in the institut3; 

1. A pre- test to assess the competence level of each participant 
before ins true t ion • 

2. Independent learning activities, adjusted to each individual's 
needs, to the extent possible. 

3. A post- test to measure growth. 

4. Certification as to what each participant was able to do 
in relation to his selected area of study. 

5. An evaluation of the effectiveness of the institute, copies 
of which would be sent to each Research Director. 

Participants of the institute numbered thirty . The Council 
of the Great City Schools assumed the responsibility for their selection 
and notification. Arr.'Jigements for transportation and hoiasing were 
handled by Teaching Research*. All participants were met at the Portland 
International Airport and motored SO miles south to Salem where they were 
lodged at the Marion Motor Hotel, Participants were assigned to cars 
by groups and driven 15 miles to the Oregon College of Education caii5)us 
in Monmouth each day. 

Ob j ectives 

The purpose of the institute was to bring research and evaluation 
personnel of the Great Cities research departments together to: 
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1, 



Provide. ixLstruction m the areas of measureanent, evaluation^ 
instructional systems, ajid proposal writing. 



2. Provide an environment conducive to independent learning 
activities and adjxisted to each individual needs' 

and 

3. Provide an assessment of the competence level of each 
participant in relation to his area of study. 

In addition to the above, terminal behaviors expected from each of 
the four areas of instruction were defined and negotiated with each 
participant and his instructor. 

Program 

In general, the instructional sequence minimized passive mas-re- 
ception and maximized active individual and smiall group involvement. 
The institute was initiated Monda)r, September 21, 1970 at 8;30 p.m., 
with an orientation during which participants and staff were introduced 
and specific goals and' procedure of the institute were clarified. 

The total group was tfien divided into four subject groups* Parti- 
cipants were given the choice of joining any one of the following: 
evaluation, instructional systems, measurement or proposal witing. . The 
group assembled in separate areas and spent the remainder of the morning 
being pre -tested and negotiating individualized performance goals for 
each member. The latter was accomplished by the instructor of each group 
sitting with each member of his group to (IJ review the results of that 
persons post-test, (2) clarify for the member, any questions he might 
have concerning the behavioral objectives established for that subject 
group, [3) identify specific interests of that member in the area, and 
(4) negotiate a set of behavioral objectives with that member which took 
into consideration the members ^pre -knowledge of the area, his expressed 
interests- in the area and reasonable limits of departure from the goals 
established for the institute. 

From these negotiations, individualized learning experiences were 
then planned for the remainder of the institute time period. With the 
exception of the final total group meeting during the late afternoon of 
the last day of the insr:itute, the institute was devoted to independent 
learning activities. 

Two ^^extra'* learning activities were also planned and carried out 
during the institute. Noon luncheons were scheduled in a convenient 
local restaurant with, provisions for all other staff members of Teaching 
Research, not involved in the institute (approximately thirty) being 
present.. - ■ 

Several lasting acquaintances between participants and Teaching 
Research- personnel have emerged from these informal interactions with- 
resulting benefits- to both. Council of the' Great City- Schools and 
Teaching Research. 
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. The second extra actiyitv^ inyolved the transportation of insti- 
tute participants and staff to the Oregon Coast on the afternoon of the 
second day to relax md enjojr the fall Beauty of the rugged coast and 
to partalce of a variety of excellent seafood. AltliougK no learning ex- 
periences directly related to the goals of the institute were planned 
as a part of the coast visit, ohvious indirect Benefits did result as 
noted in suBsequent expressions of pleasure and satisfaction from parti- 
cipants and a generally more relaxed attitude on their part during the 
rerrxiinder o£ the institute. 



TeacBers College Course One; 
M Overview 2 tKe. Theory of Evaluattoa and the 
Rational^e and Design o£ Eval' nation Studies 
Detailed Design 



LESSON ONE: Wliat is E^^aax^a^on? 



Comments to the Instructor 

The purpose o£ this lesson is to acquaint the student with the 
term "evaluation." Recall that in P?.:t I, three sets definitions for 
that tena were presented. In this lesson we introduce the student to 
these several viewpoints, merely to orient him to the materials which 
follow. It is not necessary to go into each viewpoint in detail at this 
time. We suggest that the instructor may wish to. coll attention to the 
following: 

a) Evaluation is often concerned with issues of ^IVortli". 

b) Determinations of worth may focus on program outcomes . 

c) Value judgments are involved in determining-worth. 

d) Decision maJdng is based on information about program 
operations and program outcomes; one task for the 
evaluator is to provide such infoimation, _ 

Student Reading c~ 

As a newcomer to the field of evaluation, if you were asked to de- 
fine evaluation is likely that you would define it as having some- 
thing xc do with etermining the worth, or v^lue of something. This is 
in fact the common "everyday" association with that term; To push a 
little further we might ask how one detemines wortli--ih this case, the 
worth of an ^educational program. 

' One way might be to determine how successful a program was in doing 
what? it was designed to do. A reading program which didn't teach read- 
ing might, for exaii5)le, be judged worthless. Another way of detemining 
wortfi might be to compare a program's outcomes with those of another 
programl The program with the "better" outcomes might be considered to 
have greater worth. An iJiiportant question here though is what is meant 
by the term 'better." Sometimes the issue is fairly clear cut. For 
exan5)le, if program A raises reading scores by one grade level, and pro- 
gram B by two^-other things being equal, ■program is better. However, . 
the issue is seldom so clearoit. Hoo^ does one e^ajuate reading; gains 
as opposed to "citizenship" gaohs 'as" b oiaiiges to "self -concept?' 

Such detetmnations necessarily involve' value judgn^jits — and jii racing 
an evaluation, value judgments are almost always present. 



Related to determining worth, is the process of making decisions 
based on that determination. Should a* program be continued or not? 
Should certain features of a program be changed, and if so, in what way? 
To make sudi decisions, information is needed on how program operates. 
Some who have recently concerned themselves with issues of evaluation 
in education assign to the evaluator the task of providing that infor- 
mation. In this case evaluation is viewed as the provision of informa- 
tion on \^^lich determinations of worth and related decisions may be based. 

In the materials \sMch follow, we try to give you some knowledge 
of each of these perspectives regarding the nature of evaluation. 

Suggested Exercise 

Students might be asked to ''evaluate" a course or^ experience retro- 
spectively; they might be asked to include some thoughts about how the 
. cour£;e or experience might have been improved. The instructor can use 
this opportunity to call attention to the points outlined in the above 
comments to the instructor. It seems to us particularly important that 
he (1) c;ill attention to students' value judgments concerning "worth" 
and the issue of what constitutes "inprovement," and (2) introduce the 
need for more systematic approaches to the evaluation of programs. 

Suggested Readings 

*Berlak, Harold. "Values, Goals, Public Policy and Educational Evsd- 
uation." Review of Educational Research, Vol. 40, No. 2, April, 1970. 

Glass, Gene V. 'The, Growth of Evaluation Methodology." The Evaluation 
Workshop of the Adult Education Research Conference, Minneapolis, 
Minnesota, March, 1970. 

*Klein, Stephen. . 'The Center's Changing Evaluation Model." UCLA 

Evaluation Comment , Los Angeles, Center for the Study of Evaluation, 
Vol. 2, No. 4, Jan., 1971. 

Scriven, Michael. 'The Methodology of Evaluation." Ralph W. Tyler et. 
al.. Perspectives of Curriculum Evaluation . Chicago, Rand McNally 
and Co. , 1967. ' 

Stake, 'obert E. "The Countenance of Educational Evaluation." 
' Teachers College Rbcord, Vol. 68, No* 7, April, 1967. 

*Stufflebeam, 

Suchman, Edward A. Ev^uatlve Research. New York, Russell S5.ge . Founda- 
tion, 1967.. - : • -J \ 

* CChapters 1-3 are suggested for students.) 

— L- rii; 

Tyler, Ralgh'^W; , ed. * Educational ;Evaluati^^ New Roles, New Means . 
68th Yearbook of the National ■ Society for the Study of Education, 
: Chicago, University of Chicago Pitjs^^ 

*Starred readings are suggested %of students! 



Wittrock, M.C. and Davj. ' E. "Wiley, eds. The Evaluation of Instruction. 
New York., Holt, Rinehart and IGnston, Inc., 1970. 



Wbrthen, Blaine. 'Towaxxi a Taxonomy of Evaluation Designs." 

Educational Technology, Vol. VIII, No. 15, Aug. 15, 1968, pp. 3-9. 



LESSON TWD: Input -output Models 



Coinments to the Instructor 

The concepts presented in this lesson are relevant to the first set 
of definitions presentejd in Part I above (pp. ). Those definitions 
were concerned with the relationship among inputs, processes and out- 
puts. This lesson is designed to introduce the student to these conceptg, 
particularly as they relate to education. .The purpose of the Wesson 
is to help the student grasp the conceptual significance of the concepts* 
Practical difficulties involved in employing this conceptual approach 
are not discussed until Lesson Four. 

Student Reading 

Input -output models - -in theoxy 

One way to describe an educational program- -in terms of wliat it's do- 
ing, and how, and perhaps even how well --is to view it as a system that 
uses resources in such a way as to create an inpact on students and per 
haps on the surrounding community. Resources are called inputs, iit5)act 
is called output, and thj^ way in vdiich inputs conibiiie into produce out- 
put is called process. Often the evaluator attenpts to analyse a pro- 
gram in these tenns. His concern here is one of means -end analysis. 
I-fe asks vMch' means produce which ends . This concern is embodied con- 
ceptually in the following input -process -output model. 



Resources 


are 


or 


assigned 


Inputs 


to ^ 



Processes 
or 

Programs 
or 

Activities 



wliich 
contribute 

to^ 



the 
attainment 
of 



' Ends 
or 

Objectives 
or 
Outputs 



Weans 



Ends 



By applying this schema to an ongoing pi^ogram, the evaluator can 
ascertain whether a program is acconjjlishing What it is si5)posed to 
accomplish Ci-^., whether the* actual ends are similar to the ends which 
were envisioned at program inception) , and vMch process or input yari- 
ables account for success-or lack of success. 

Ultimately, toe purpose of this ^ analysis is to find the 

best ineans for acconplishihg.certain: desired ends. Oiie uses this type 
of analysis in askijig, for example, whether reading program A or reading 



program B provides the best means for acconplishing certain given 
objectives. Or one might ask how reading program A might be changed in 
order to maJce it more effective. Let us turn here to a fuller dis- 
cussion of the components of this model, with specific attention to the 
content of each component in education. 

(a) Ends (goals, objectives ^ outputs) 

All planned action is undertaken with a view toward accomplishing 
some end.. Statements or descriptions of these ends may range from very 
broad general" statements on the one hand to very specific operational 
definitions on the other hand. The term '*goal" is generally associated 
with the former, and ther "objective*' with the latter. Theoretically, 
then, one can view ends as ranging from general to specific, from ab- 
st-^-^ct to concrete; goals fall at one end of the continuum and t5bjectives 
Csi.me educators would say '^behavioral objectives") fall at the other 
extrc^me. That is to say, ends may be stated with varying degrees of 
specificity. They may range from very specific behavioral objectives, 
to ends idoich lack behavioral specificity but vfliich are more concrete 
than goals, to abstract goals. Thus the following might be illustrative 
of one type of ends continuum for education:!/ 

Increased number of people able to 
function well as meii4)ers of society. 

Increased number of people able to function 
/ as literate members of society. 

Increased conprehension by a number of high school 
graduates pn a sample of newspapers, magazines 
and other materials typical of those encountered 
daily by the average citizen. 

Increased conprehension by a number of students on a 
reading conprehension test. 

Increased performance by a number. of students on a reading 
proficiency test. 

(Outputs and Objectives) 

The term "output-* is yet another term. used to- designate ends. Usual- 
ly the terms "goals" and "objectives" describe planned "targets," while 
"outputs" refer to program resu] , achieved by aiming at planned tar- 
gets. In other words, at the b€ inning of a program, ends are stated as 



1/ Suchman, op, cit. , p. 101. 
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objectives or targets; at the end of a program isflien ends ai ^ actually 
achieved, wr- c;ill tliem outputs. Outputs, like objectives are usually 
described ir) --pocific concrete ternis. For our purposes here we shall 
define object i.v3s as specific statements of program targets, and out- 
puts as specific statements of program results. Ideally, then, outputs 
will be the same as objectives. The following are just some exaii^les 
of educational ou^^'-it::, (objective) propos(:d in the literature: 

leamiiig in standard subjects 

achievement in specie:! subjects 

improvement in tti? ability to reason 

stimulus to intellectual curiosity 

stimulus to creativity 

development of social poise 

reduction of emotional disturbance 

improvement of physical health 

various aspects of socialization including prevention of 
delinquence 

opportunity option of further ediicationl/ 

Some educators insist that objectives (outputs} be described in 
more specific terms than the above. The most extreme version of this 
position is that of the Be'^avioral objectivists who insist that a well- 
written objective must specify all of the following: 

1) What it is that a student who has mastered the objective will 
be able to do, 

2) under what conditions he will be able to do it, 

3) to what extent he will be able to do it. 2/ 

IVritten to fulfill these requirements, behavioral objectives are 
highly specific statements. The following exairple of a behavioral ob^ 
jective demonstrates the degree of specificity that is required: 



1/j.A. Kershaw and R.N. McKem, Systems Analysis and Education , 
Santa Monica, Calif., The Rand Corp., Oct. , 1959, p. 8. 

Jesse Burkhead, Thomas G. Fox, and John W. Holland, Input and 
Output in Large -City High Schools, Syracuse University Press7 1967, p. 24 -26. 

2/ Thorwald Ebensen, 'Writing Instructional Objectives," Phi Del^ 
Kappan , 48:246, Jan., 1967. 
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Gi\en ten ^alrs of short prose passages^-each pair ha'^'-ing 
one selection by Ernest Hemingway and one by a rL.ffor.ent 
author the student If- .^blc, with, at least 90 pevcent 
accuracy, to clioose the 10 selections writtv°n by Heniir.orway 4_/ 

(Goal) 

The term ^'goal /' on the other hand, may be defined as a general 
statement describirig an abstract state which an individual or group 
would like to at^aLi. A^i educational goal is thus a general statement 
about kinds of education that are desirable, or the kinds of things 
that education should accomplish. An exaii^le of this kind of statement 
is a goal proposed by the U. S. Office of Education. 

The overall purpose of the educational system is to foster 
self-evaluation and self-fulfillment--to enable each in- 
dividual to function. successfully and to the fui.l extent 
of iiis abilities in both the economic and non-economic 
sectors of society. Each individual must have opportuni- 
ties for both occupational training and for other kinds of 
education for self -development , including the ability to 
make informed and constructive decisions .2/ 

Although it may not be made explicit in a statement of educational 
goals, often such goals are derived from societal goals, or notions a- 
bout wlaat is desired for the individual and society. That is to say, 
educational goals are often believed to contribute in some way to social 
and individual goals or to the things that are desired for individuals 
and society. Thus equality of educational opportunity is believed to 
contribute to increased social equality, and civics education is believed 
to produce an informed electorate. Whether or not such is, in fact, 
.the case, is the subject of debate and some study. As a contributor to 
this study it is important for the evaluator to be aware of the context 
within which educational goals are postulated. Are the goals compatible 
with that context? And if not, what are the implications of the 
divergence?? 

(h) Inpu^,s or Resources 

Goods^ and services --materials and supplies and human effort- -are 
required to produce desired ends. Inputs are the goods and services- - 
human and ncn-humaji- -which are used to produce those ends. The/e are 
a variety of ways to categorize inputs for education. One simple taxonomy 
is offered here merely fo^ purposes of illustration: xi^' 

Student Time 

Personnel time 



J/ Ibid., p. 247 

jfu.S. Dept. of Health, Education, and Welfare, Planning Program: i jig 
Budg.eting , Washington, D.C., April 17, 1968, p. 21 
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aciministrative 



teaching 
clerical 
mintenance 

auxiliary (guidance, health, library) 

Buildings and equipment 1/ 

(c) PrograjTis or processes 

Educational programs or process are the actual activities carried 
out in schools or educational settings. The following description o£ 
those activities is provided by a prominent writer on ^ucational eval- 
uation; 

Transactions are the comtless encounters o£ students with 
teacher, student with student, author with reader, parent 
with counselor- -the succession o£ engagements which comprise 
the process o£ education. Examples are the presentation o£ 
a £ilm, a class discussion on tlie margin o£ a term paper, 
and the administration o£ a tost, V 

More technically, within the context of an input -output model,* --programs 
or processes may be viewed as "transformations^' or ''mediating factors, ''r/ 
That is to say, they mediate or link input causally to output: or they 
may actually transform an input into an output. 

Characteristics of the programs include proportions and amounts of 
inputs used and the dynamics of their interaction over time» It is this 
latter characteristic which makes it so difficult that the way in which 
a teacher interacts with a student (if he is authoritarian instead of 
permissive for example.) influences the student's learning, then it is 
insufficient to describe the program as involving just any teacher-stu- 
dent interaction; the quality of the teacher -student interaction must 
be specified. Implications of the problem of describing the dynamics 
of programs are discussed more fially later in this 'course. 



V Burkhead op . cit. . p. 24-26 . 

1/ Robert E. Stake, "The Countenance of Educational Evaluation," 
T. C. Record, 68:528, April, 1967, 

3/ - 

— Marvin C. Alkin, "Behavioral Objective Specifications in Eval- 
uation: Relevant and Irrelevant, "Western Regional Conference on 
Testing Problems, Princeton, N.J. , Ed; Testing Service, May, 1969, p. 43 • 

Gainr Andrew':and Ronald E. Moir, Iriforination-^^ Systems in 

Education, Itasca^, 111. , F,E. Peacock Publishers, inCv, iy/0. 



Suggested Exercises 



Students might be presented with a sljnple list of '^scrambled** in- 
put process and output variables, and be asked to decide whcit kind of 
variable each one was. Cm such list is included in the Provis train- 
ing materials. Or the instructor may easily develop his ov/n list. It 
is important for purposes of this lesson to keep the exercise single. 
For the moment, the student is asked merely to grasp the conceptual 
differences among resources, processes and output. Practical difficul- 
ties in trying to sort out real situations are brought oi;t in later 
lessons. If questions of feasibility are raised, the instructor should 
use them to lead into these later lessons. 

Suggested Readings 

*Alkia, Marvin C. ' Uenav7,oral Objective Specifications in Evaluation; 
Relevant or Irrelpv^nt Western Regional Conference on Testing 
Problems, .Prince 'or^. Educational Testjjig Service, May,. 1969. 

*Alkin, Marvin C. Towards and Evaluation Model: A Systems Appioach . 
Center for the Study of Evaluation of Instructional Program, 
Los Angeles, University of Calif,, Dec, 1967. 

Burkliead, Jesse, Fox, Thomas and John W. Holland. Input and Output 
in Large City Schools . Syracuse, Syracuse University I'ress, 1967. 

*Ebensen Hion/ald. "Writing Instn.:ctional Objectives." Phi Delta 
Kappan , vol. 48, No. 5, January, 1967, pp. 246-247. 

*Kenshaw, J. A. and R.N. McKean. Systems Analyses and Education . 
Santa Monica, The Rand Corporation, October, 1959 

LESSON TliREE: I nput -Output Models as Systems 

Comments to the Instructor 

By this point the student has already been exposed to some concepts' 
of systems. The pu:rposes of this lesson are to introduce the concept 
of **system** explicity, to show how educational programs may bs described 
as systems, and to suggest that systems concepts are useful to program 
evaluation and program improvement*. 

Student Reading 

No doubt you are familiar with the term **system." Recently there 
has developed significant interest in using a formal concept of "system** 
for describing, analysing and learning to manipulate certain kinds of 
social programs, institutions or situations. 



^Starred readings are suggested for students. 
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The input -process -output model may in fact be thought of as a 
system. That model is enconpassed by a definition which describes a 
system as '*a set of interrelated facors that are used together to pro- 
duce an output output."!/ Thus inputs and processes combine to produce 
an output. 

Important in the definition of systems are inter -dependent or in- 
terrelated, a change in one element brings about change in others. Thus 
by discovering the nature of dependencies or relationships, one can 
leam how to manipulate elements of a system in order to achieve desired 
effects or changes in effects. In education, for example, the decision- 
maker may warit to know how to change some process element in order to 
bring about an improved program output. 

Educational programs, indeed educational systems, may be described 
and analyzed in terms of an input-output systems model which calls 
attention to the nature of relationships among system lents. The 
model provides an aid to planning program implementation, evaluation, 
and to the many areas of decision -making that accompany these activities. 
Analysis of the system indicates to the decision maker what kinds of 
changes among inputs and processes may bp necessary if desired outputs, 
or objectives, are to be met. Modification of objectives is also per- 
mitted within this analytic framework. Later in the course we will dis- 
cuss more fully how the input -output ^systems model can be used to assist 
the decision-maker or administrator as he carries out his program plan- 
ning, evaluation and implementation. 

The systems and its environment 

One may conceive of the entire universe as a system whithin a 
system within a system. The particular system that is selected for 
analysis depends of course on the problem to be studied. It is impor- 
tant to carefully define the boundaries of the system to be studied; 
often, particularly in edication such definitions are, of necessity, 
quite arbitrary. In defining a particular system as the one of immedi- 
ate concern one must also bear in mind the factors in the broader en- 
vironment, external to that system, that may in fact produce an iii^^act 
on the system. If. a factor is likely to produce a significant impact, 
one may want to include it within the defined system. It not, such 
factors are defined as part of the "environment," which surrounds the 
system, or as part of "external systems ,"2/ 

In education, the environment is often viewed as encompassing non- 
school, commuTiity, family socio-economic or political factors. The 
system itself js often definced as a particular .school or set of schools. 
However, one might just as correctly define a classroom as a system; 



—'Kershaw and McKean, op. cit . , p. 2. 
^^Alkin, CSE Report, p. 43 
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adjninistrative and other school characteristics as the environment. Or 
ome might define as a system even more specific interests such as a 
particular curilculiim or instructional sequence. 

Suggested Kxercise 

Students might be asked to describe some sinq^le mechanical system, 
establis^iing the boundaries between the system and its environment, and 
describing how system elements are related. They might then be asked 
to sketch out the same for some social system, again being careful to 
describe the relatedness of elements. Specific attention should be 
drawn to the effects of manipulating system elements. Differences be- 
tiveen descriptions of mechanical systems and social systems should be 
noted . 

Suggested Readings 

'^Andrew, Gary and Ronald E. Moir. Information-Decision Systems in 
EducatioD Ithaca, III., F. E, Peacock Publisliers, Inc,^ 1970. 

^Carter, Launor, The Systems Approach to Education - The Mystique and 

the Reality , Santa Monica, S^'^tem Development Corporation, Jan, 27, 1969. 



LESSON FOUR: Input -Output Models in Practice 

Comments to the Instructor 

This lesson is designed to acquaint the student with some of the 
practical realities of trying to apply input-output systems concepts to 
actual programs in education. By and large the problem of application 
is not that the concepts are inappropriate, but rather that they are 
difficult to operational ize. It is likely that students will already 
have begun to voice concern about some of the issues presented here; if 
so the instructor may want to introduce this lesson by recalling some of 
those concerns . 

Student Reading 

The input -output model that we have been studying does offer a 
neat, logical way to describe what goes on in a particular program. But 
try to describe an education program with which you are familiar in in- 
put-output terms. Can you reduce the objectives' to measurable variables? 
Can you even begin to identify and describe significant process varia- 
bles? 

These are some of the problems encountered in trying to use an in- 
put-output systems model. In the next few pages, we'll examine carefully 
tlie features of this model which limits its application to real problems 
in education. 



^Starred readings are suggested for students. 
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Developing operational statements of objectives 



Unless an output or objective is described in ver>^ concrete terms, 
there is no way to Inow vdaether or not it actually occurs. For example, 
it is not enough to say that kids should be able to read at the end of 
a program. If you really want to know the effects of the program, you 
need to know whe\:her the kid staring at a wi-itten page is reading or 
not you must specify behaviorally what you mean by reading. 

Recent concern for developing such behavioral specifications has 
had many educators to advocate the use of behavioral objectives. As 
we have seen (pp. behavioral objectives must be stated in highly 

specific behavioral terms; the most extreme position regarding the de- 
gree of specificity required, insists that a well-written behavioral 
objective must specify what the student should be able to do, under what 
conditions, and to what extent.!/ 

Use of behavioral objectives had been advocated by a few educators 
since the early 1900's,2/but it wasn't until the 1960 *s that such ad- 
vocacy began to have any effect on the way educators stated their objec- 
tives. The impetus for the ciirrent wide.spread interest in behavioral 
objectives came undoubtedly irom the development of new instructional 
systems which required precise formulation of outcomes, and from federal 
funds. As a result, a number of states have issued state-wide instruc- 
tions for evaluating programs in terms of behavioral objectives, and 
the State of Florida now requires all teachers and school systems to 
prepare behavioral objectives for their own instructional areas. V 
Furthermore, an instructional objectives exchange has been established 
at the Center for the Study of Evaluation at U.C.L,A. Educators are 
encouraged to submit objectives as well as to use those already developed 
by. the center. The center provides a variety of collections of objectifies. 

On the surface this appears to be a very promising development in 
that for the first time, educators are beginning to do what systems 
analysts say t?ie must do- -that is, to describe output in concrete, mea- 
surable terms. Indeed, it probably is an irrportant advance, but there 
has been some very heavy and iiT5)ortant criticism levelled against cur- 
rent uses of behavioral objectives. Most significant perliaps are the 



2/Ebenson, op. cit. , p. 246. 

— James .W. Popham, Inst ructional Objectives Exchange:' a Project of 

the Center for the Study of Eval u ation , Los Angeles, U.C.L.A. Graduate 
School 

3/Director H.H. McAshan, Writing Behavioral Objectives, Gainesville, 
Fla., Fla. Ed. Research 5 Dev. Council, Col. of Ed., Revised Sept., 
1969, p. 3. 



ERLC 



-85- 



the niamerous charges!/ that the use of behavioral objectives tends to 
reinforce most easily meacared behaviors , at the expense of behaviors 
>Mch are less easily measured. At the extreme, this may result in a 
situation in whicli ease of measurement rather than educational value 
determines vdiat outputs are produced. This position is well -stated by 
Myron Atkin v4id first describes multiple lean.ing outcomes and the dif- 
ficulty of measuring all of them; he then states: 

If identification of all worthwhile outcomes in behavioral 
terms comes to be commonly accepted and expected, then it is 
inevitable that, ever time, the curriculum will tend to' em- 
phasize those elements vMch have been'flius identified. Im- 
portant outcomes \4iich are detected only with great diffi- 
culty and which are translated only rarely into behavioral 
terms tend to atrophy. They disappear from the curriculum 
because we spend all the time allotted to us in teaching 
explicitly for the more readily specifiable learnings to 
which we have been directed. 2/ 

Further, it is charged that the incautions use of behavioral objectives 
will stifle idiosyncracy creativity, social orientation and perhaps even 



~" See for exanple : 

Benjamin S. Bloom, 'Toward a Theory O): Testing Which Includes 
Measurement -Evaluation -Assessment, " Occasional Report No. 9, Los 
Angeles, U. of Calif . , October, 1968 

Lee J. Cronback, ^'Course Improvement Through Evaluation," 
T.C, Record , 64:672-683, May, 1963. 

Henry S, Dyer, 'The Discovery and Development of Educational 
Goals Proceedings of the 1966 Invitational Conference on Testing 
Problems, Princeton, N.J., Educational Testing Service, 1966, p.. 12-23. 

Elliot W. Eisner, 'Instnactional and Expressive Educational 
Objectives: Their Formation and Use in Curriculum/' in James W. Popham, 
Elliot Eisner, Howard J. Sullivan and Louise R. Tyler, Instructional 
Objectives, AERA Monogra]|jh Series in Curricid.um Evaluation , Chicago, 
Rand McNally and Co., 1959, pp. 1-8 and 13U-132 

Samuel Messick, 'The Criterion Problm in the Evaluation of 
Instruction: Assessing Possible Not Just Intended Outcomes,'' Center for 
the Study of Evaluation of Instructional Programs, Report No. 22, Los 
Angeles, U. of Calif., May, 1969. 

?i/}fyron J. Atkins, "Behavioral Objectives in CurriciiLum Design: 
A Cautionrry Note," Science Teacher , 35:27-30, May, 1968. 
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feel tag of self-wortli.i/ 



Use of standardized tests 

Often standardized tests are used as an indicator of output. Tests 
liave some u^e in that thjey help to indicate how students and schools 
are doiiig vis-a-vis o titers. However, once again, they axe of question- 
able adequacy for tb^e measurement of educational output. Consider just 
some of the criticisms of standardized tests: 

i] Tests do not really measure gains in skill .iz a result of 
instruction. Some charge that they are much tJie same as 
intelligence tests and <'is such do not provide a valid 
raeasui-e of the effects of schooling. Otiiers point that 
many tests are weighted h^eavily in terms of verbal abili- 
ty and therefore measure fhat rather than achievement. 2/ 

2) Tests reflect the influence of coacMng by teachers vho 
are "teaching for the test." V 

3) Tests place the child in an anxiety-provoiJjig situation; . 
for the youth lacking self-confidence, iJne situations^ 
my detract significance from his performance^/ 



— 1/see for txaiq^le: 

Henry S. Dyer, op. cit. 
Elliot Eisner, op. cit. 

Pauline S. Sears and David H. Feldman, Changes in Yonng Children's 
Classroom Behavior after a Year of C onp u ter Assisted ^Instruction: jST" ^ 
hxplorator>^ Study, Research MemoranBum. 

Michael A. Wallach, '^Creati^/ity and the Expression of Possibili- 
ties,*' Cre ativity and L earning , Jerome Kagan, ed. , Boston, Houghton 
Mif f 1 in Co., 1967. 

2/ 

— See for exanple: 

R.W. Skager and L.A. Broadbent, ' 'Cognitive Structures and 
Educational Evaluation, Center for the Study of Evaluation of Instructional 
Prograjus, Occasional Report No. 4, Los Angeles, U. of Calif., July, 1968. 

A.G. Wesman, 'Intelligence Testing,'' American Psychologist, 23: 
267-274, 1968 

3/ 

~ See for examp.'^e: 
Mi:riam Wasseman, "Planting Pansies on the Roof," CA Critique 
of Now N.Y. City tests Reading), The Urban Review , January, 1969. 

V 

See for exanple : 

Miriam Wassexm, op. cit . 
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4) Reading tests items contain content unfamiliar to the 
ghetto youngster so that he is faced with the task not 
only of reading, but of trying to make sense out of 
words that are alien to him.l/ 

5) Tests do not measure true intellectual competence in that 
tJiey neglect ^'depth, connectedness, and applicability of 
knowledge. 2/ 

6) Tests may reward the retentive, docile student who seeks 
the single "correct'^ answer, and penalize the more crea- 
tive student who seeks nove] approaches to problems. 3/ 

7) Tests damage and humilate the- disadvantaged child parti- 
cularly by projecting society's inadequacies onto him.V 

Use of behavioral objectives and standardized tests by tiae evaluator 

- Miat is tho evaluator to make of this dilemma? On the one hand he 
needs to specify as precisely as possible those outputs he is looking 
for and he nees ways to measure them, Otherwise he won't "know'' them 
even when lie "sees" them. On the otlier hand, he must avoid using 
inaccur age, incomplete measures as they can make his analysis inaccurate 
and incomplete. 



—'See for exaii^le; 



Cronbachy. op. cit . 

R. Glaser and R.C. Cox, "Criteria-Reference Testing for the 
Measurement of Educational Outcomes," Instruc t ional Process and Media 
Innovation > R. Weisgerber, ed. , Chicago, RandTicNally and Co., 1968, 



R.E. Stake,- * Toward Technology for the Educational Programs,^' 
Perspectives of Curricul u m Evaluation , R.W. Tyler, R.M. Gagne and M. 
Scriven, eds., Chicago, .Rand McNally and Co., 1967, pp. 1-12. 

— See for exan^le : 
J.K. Getzels, "Non-IQ, Intellectual and Other Factors in College 
Admission," T he Tyra nr y of Testing , Baiiesh Hoffman, ed. New York, 
Crowell" ColTier Publishing Co., 15(52, p, 101. 

Banesh Hoffinan, ed. , The Tyrailny of Testing > New York, Crowell- 
Collier Publishing Co., 1962. 



Ibid. 





See for example : 
lAEasserman, d p. cit . 



Our solution is only partially satisfactory. We suggest that some 
reliance must be placed on statements of behavioral objectives and qji 
standardized tests. The evaluator may also wish to develop some of his 
own tests or survey instruments. However, quantitative analysis based 
on these tools must be viewed as only part of a conq^lete evaluation. 
Equally iji^ortant are ( 1) co mmei^ts on tlie relevance of the derived quan- 
tities, C2} qualit'atl^W analysiss of portions of the prograjn that can 
not be adequately characterized in quantative terms at the time of 
the evaluation. 

Selecting critical process variables for aiialysis 

The input -output model requires the evaluator to analyze how pro- 
cess liiiks inputs to outputs. And yet, vtfiat is tlie educational process? 
And which variable are most iji5)ortant? Quality of curriculum? Teacher 
experience? Teacher personality? Pupil motivation? Pupil health? 
There is surprisingly little information on rJiese questions. 

One perplexing problem has to do with sorting out the efftcts of 
school processes from thiB effects of other factors. Much of the con- 
troversy over the Coleman report, for exairple, centers on just this issue 
What effect do schools have? Techniques have not beer* derived to iden- 
tify those e?7ects with any degree of validity. If we look at the in- 
structional process--the interactions of teachers and materials with 
students--we find that very little information exists about the nature 
of that process. Are small classes more effective than large classes-- 
for whom, under what conditions, and at what cost? Is heterogeneous 
grouping superior to homogeneous groiqjing? How do discussions compare 
with lectures? How do live and television presentations conpare? Under 
what conditions may non-directive instruction be more effective than 
teacher -centered instruction? What are the relationships between teacher 
personality and effectiveness- -and betwen student personaility and 
learning?!/ 

Remember also that the instructional process is only part of the 
educational setting. Some variables outside of the immediate school 
setting may have an in^ortant effect on student learning. But what these 
variable are and how they operate has not been determined. What is the 
effect of home characteristics? What is the effect of socio -cultural 
.characteristics? What is the effect of teacher unions and supervisory 
associations? What is the effect of community organizations? 

To isolate and begin to understand the effects of process varia- 
bles, it seems to us, again, that qualitative field-oriented methodolo- 
gies are needed in addition to the more quantitative methodologies. For 



Marvin Bressler and MelviJi M. Tumin, Evaluation of the Effective- 
ness of Educational Systems Final Report, Cooperative Research Project 
No. 6-2aZ3, Washington, D.C., U. S. Office of Education, Dept. of Health, 
Education and Welfare, E-4, p. 13. 



it would seem that only by going to the field and looking at what's hap- 
pening can one ~tnily begin to understand the nature of the educational 
process. 

The course which w 3 have designed to conrplement this basic course 
on evaluation will be giving you practice in doing just this- -in going 
to the field, looking and listen±ng, and reporting and analysing what 
you have seen and heard. 

Suggested Exerc i ses 

At this point the student should practice sorting a coniplex pro*- 
gram description into inputs, processes and outputs. If students are 
involved in an ongoing, evaluation, they may want to try organizing that 
program in teims of the input output mode. There are several other 
sources for existing materials \Alch describe programs and these may 
also be used for these purposes. Materials which are being prepared by 
Malcolm Provus, and documents for the Experimental Elementary Program 
available through Teachers College are suggested sources. These materials 
are described in greater detail in the Appendix. 

Students should be encouraged to note the problems they encounter 
in trying to sort the program into input-output categories. Ideally 
they will note the following : 

1) Program goals may not be ijnmediately obvious from program 
documents. They may have to be "pulled" from program 
descriptions or irom interviews with program administra- 
tors and staff* Further, various participants are likely 
to have different views regarding program goals. 

2) Statements of program goals are often stated at mixed le- 
vels of specificity. In such cases, it is inportant to 
develop a hierarchy ranging from specific to abstract. 
The specific statements serve as the targets against 
which program performance is measusred. - 

3) There will be some outputs \diich are not easily measured. 
Provision should be made to include qualitative statements 
of these outputs in the program description. 

4) Besides the process variable which are mentioned in the 
program description there may be others which students 
feel may be in5)ortant. They should be encouraged to 
list and justify those additional variables which they 
feel may be important. 

5) Programs take place over time. If students feel that the in- 
put-output model needs to be adapted to reflect the sequential 
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features of a prograjni, they should be encouraged to do so.V 
Suggested Readings 

*Atkins Myron J. "Behavioral Objectives in Curriculum Design: A 
Cautionary* Note." Science Teacher , 35:27-30, May, 1968. 

Bloom, Benjanjji S. * Toward a Tlieory o£ Testing which includes 
Measurement --Evaluation-Assessment." Center for the Study of 
Evaluation of Instructional Programs, Occasional Report No. 9, 
Los Angeles, IMiversity of Calif., October, 1968. 

Bressler, Marvin and Melvin Tumin. Evaluation of the Effectiveness of 

Educational Systems, Final Repo rt". Cooperative Research Project 
No. 5-2023, U.S. Office of Education, 1969. 

Cronbach, U.J. ''Course Improvement through Evaluation/' Teachers 
College Record, Vol, 8, May, 1963, pp. 672-683. 

*Dyer, I-fenry S. 'The Discovery and Development of Educational Goals 
'Procee dings of the 1966 Invit at ional Conference on Testing 
ProFf ems^ Princeton, Educational "testing Service, 1966, pp. 12-23. 

*Eisner, Elliot W. "Instructional and Expressive Educational Objectives: 
Their Formation and Use in Curriculum-" in James, W- Popham and 
Elliot Eisner, Instructional Objectives . AERA Monograph Series 
in CurriculiSTii Evaluation. LMcago, Kand McNally and Co., 1969, pp, 
1-18 and 130-132. 

^Lortie, Dan C. ''Rational Decision-^4aking: Is it Possible Today?" 
The EPIE Forum , 1967, pp. 6-9. . 

*Messich, Samuel. The Criterion Problem in the Evaluation of Instruction : 
Assessing Possible Not Just Intended Outcomes . Center for the Study 
of Evaluation or Instructional Programs, Report 22, Los Angeles, 
University of Cal., May, 1969. 



— This is one of the problems we are working with in our town 
studies; Professor Frank Smith has made iit^Dortant contributions to 
our understanding of the problem. For example, we are developing an 
''input-conversion-process-output" model, conversion being the trans- 
formation of certain input variables into the form required for 
process transc^ctions. Teacher orientation might be one such example 
of a conversion. We are also developing ways to account for the 
order in which program elements need to be assembled. This is, of 
course, similar to the work being done with PERT, our focus being 
that of evaluation. 



^Starred readings are suggested for students. 
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*Popham, James W. Instructional Ob - fect^^ Exchange; a Project 
of the Center for the Study" of Evaluation. Los Angeles , 
UCLA Graduate School of Education. 

.« Stake, R.E. "Toward a Technology for the Educational Programs," 

R.W. Tyler and Gagne eds. Perspectives of Curriculimi Evaluation . 
Chicago, Rand McNally, 1967, pp. 1-12. 

*nVasserman, Miriam. '^Planting Pansies on the Roof." CA Critique of 
howN.y. City tests reading) The Urbaii Review , January, 1969. 

LESSON FIVE: Informing the Decision Maker; the Evaluation Role in Per- 
formiag"~a Systems Analysis i ] ^ 

Comments on the Lesson 

Recall that one definition of evaluation had to do with generating 
information for use by the decision -maker • Recently techniques of sys- 
tems analysis have been promoted as particularly relevant to the deci- 
sion-making function. The success of these techniques in space ex- 
ploration and the military, has lead many to suggest that they are simi- 
larly useful in education. This lesson introduces the student quite 
briefly to the steps of a systems analysis, and the evaluators, function 
in that analysis; then we introduce the student to the CSE model to bring 
out the practical consequences for the evaluator in implementing a sys- 
tems analysis in education. 

The lesson is structured on tiie asjsumption that students will have 
available the CSE training materials, published by McGraw Hill. Those 
materials are designed for tnteisive use over two days. The instructor 
may of course restructure the timing of his own presentation (see 
pp. above). We suggest further that large blocks of time be de- 

. \Dted to discussing those issues dsid problems which we suggest in our 
own student readings. 

Student Reading 

Recall that in the first lesson it was pointed out that some eval- 
uators view their function as that of providijig information to the deci- 
sion-maker. That dec is ion -maker is concerned with planning, implement- 
ing and evaluating programs, and as such he needs certain kinds of 
information at each step in this sequence. It is often proposed that one 
model of the tas'^c of the decisionsnaker in planning, implementing and 
evaluating a program is the model of systems analysis. Let's look for 
a moment at that model and consider its implications for the evaluator. 

Systems analysis is really little more than a formalized model of 
a common sense approach to problem solving. One commentator on the sub- 
ject has for exaji^le, described systems analysis as . .a logical 
step-by-step approach to problem solving which we use continually even 
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though- we perform many o£ the steps unconsciously."!/ Systems analysis 
is sometimes described as a set of steps involved in the design of a 
solution to meet an expressed need^ A subgroup of the National Security 
Industrial Association, considering ttie application of military and 
space technology to educational problems, summarized these steps as 
follows : 

1. State the NEED-*that prohlevOi which initiates consideration 
of an education/ training system as a potential solution. 

2. Define the OBJECTIVES- -the terminal capability desired of 
students after having successfully completed a learning 
experience- -which will contribute to satisfying the need. 

3. Define the CONSTRAINTS, or real -world limiting conditions 
which must be satisfied by any acceptable system designed 
to attain the educational objectives. 

4. Generage ALTERNATIVES, or candidate systems which could 
achieve the objectives. 

5. SELECT the best alternative by systematic evaluation of 
alternatives in terms of objectives and constraints. 

6. IMPLEMENT, or adopt the selected alternative to meet. the 
specified objective. 

7. EVALUATE the implemented objectives by detematng* the 
conformance or discrepancy between all of the objectives 
initially specified and the performance that was actually 
obtained. 

8. MODIFY the designed learning system based on deficiencies 

in meeting the objectives as detemined through evaluation.2/ 

These procedures lie at the heart of all the many systems analytic, 
managerial and decision making techniques. Notice that evaluation is- 
part of this process arid that the evaluative function, step 7, is 
dependent on the implementation_,of the preceding 6 steps; evalioation in 
the systems analytic sense requires a prior detemination of objectives 



ERLC 



Instruction 



^Henry Lehmann, 'The Systems Approach to Education," Audiovisual 
ction, ^ Feb., 1966, pp. 144-148. 



--^Lehman, op. cit ., p. 145'-148. 

Launor F. Carter, The Systems Approach to Education- -the Mystique 
and the Reality, Santa Monijca, Calif • /syst^ Development Corporation, 
Jan. Z7, 1969, pp. 2-3. 
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based on needs assessment, and on rationalizing and in^lementing a 
particular program intended to meet those objectives. For this reason, 
evalimtors are encouraged to view their function as being related to 
this entire sequence or procedures used in systems analysis or decision • 
liialcing. The evaluator's function in this sequence is integral to the 
specification of program goals and their requisite programs. While the 
administrator may, arid usually does, reserve the right to malce actioal 
decisions regarding goal and program selection, the evaluator too per- 
forms an important function in this regard; for his is the skill re- 
quired in stating goals and describing programs and alternatives in terms 
which are subject to evaluation. 

Regardless of how neat and logical all of this seems, bear one 
point in mind as you procede through this lesson. The systems analytic 
model is an ideal model. It describes how decisions should be made un- 
der ideal circumstances- -circumstsnces in \>W.ch the dec is ion -maker is 
disposed to decide rationally, and in which key variables can be identi- 
fied and measured. You are already aware of some of the problems in 
identifying and measuring key variables in education. No doubt you are 
also aware that decision-malcers may not plan rationally for programs, 
and may ask for an evaluation after the programs are fait accompli . 
Evaluation can still be conducted but it will not enr:oin)ass all the 
stages of the ideal model. ^ 

The CSE Model 

The Center for the Study of Evaluation at U.C.L.A. has developed 
an evaluation model, based on the systems approach, which specifies the 
kinds of evaluative information and assistance that are required by the 
decision-maker in the sequential stages of planning implementation and 
evaluation- -roughly tlie same stages as those involved in a systems 
analysis. The Center defines evaluation as follows: 

Evaluation is the process of ascertaining the decisions to 
be made, selecting related infonnation, and collecting 
and analysing that infonnation in order to report summary 
data useful to dec is ion -makers in selecting among alter- 
natives.!/ 

To provide information relevant to the various stages of detision-making 
the Center has developed the following five phases of evaluation acti- 
vities : 

1) Needs assessment 

23 Program plaiming 



■-'Alkia, CSE Report, op cit . 
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3} Ijnpleinentation evaluation 

4) Pi ogress evaluation 

5) Outcomes evaluation 

The exercise at the end o£ this lesson will take you through each of 
these phases in some depth. In the next pages we'll discuss each 
phase Ijriefly fx'om systems perspective. This Hiaterial should serve to 
supplement the exercise which follows. 

Needs assessroent; setting program goals and objectives 

From a systems perspective, activities are undertaken with a view 
toward accomplishing certain ends. Economics, as well a<v coinmon ense, 
mandate that not all ends car. be met, given limited time and resources. 
For this reason, it is important to decide \Aich ends have highest 
priority, and to establish programs to meet those ends. This is the • 
first step of a good systems analysis; it is also the fitst step in the 
CSE model. Essentially what is involved is operationalizing statements 
of goals and assigning priorities to those various objectives. 

The evaluator is likely to encounter two major problems in con- 
ducting a needs assessment. The first has to do with developing ways to 
operational ize goals--with developing statements of objectives which 
are inclusive and which can be observed or measured. You have already 
encountered this problem in Lesson Two. 

A second problem has to do with how ycu go about deciding which 
objectives are most important. Whose values .do you use? Those of the 
administrator? teachers? community members? If you use some combination 
of individuals, how do you weight their preferences? In the CSE exer- 
cise on Needs Assessment you are provided with weighting procedures. 
As you use the procedures, recall a situation with which you are already 
familiar and try to determine whetlier the same kind of weighting proce- 
dures would be appropriate in that situation. 

Program Planning 

Ideally, before a program is undertaken- -particularly a new, inno- 
vative program- -the administrator engages in some sort of planning. He 
decides vdiat ends he would like to attain, and tries to determine the 
best means for attaining those ends. The literature contains many de- 
finitions for the term ^'planning'* although they coimonly imply the type 
of means -ends conceptualization embodied in the input -output model dis- 
cussed in Lesson Two. Consider for example the following devinition 
of planning: 

Plannii^ is the process o£ preparing a set of decisions 
for action in the future, directed at achieving goals 
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by optimal means. V 

Tliis definition, sijnilar to many calls attention to the following 
important characteristics of planning: 

1) Planning is goal- oriented. That is to say, planning is under- 
taken with a view to achieving one or several ends. These 
ends or goals are the raisbri d^^etre of any plan; they are 
integral to the planning process. 

2) Pl anning focuses on available means, or ways of combining 
resources . As we have seen, there are often several means 
by vrtiich ends may be reached. Planning assesses these 
means and specifies which among all, are to be implemented 
in order to achieve ends. 

3) Planning is concerned with deciding which means is ''best .^^ 
Given a variety of means by which ends may be achieved, 
the planner must select which means are "best." An 
in^ortant criterion in assessing \ihat is best is that of 
optimality or efficiency; operationally this means that 
often the planner is concerned with findiag the least- 
cost means by which the stated ends may be achieved. 

(Of course, given the present lack of knowledge in 
education about the relation between means and ends^ 
the planner's task more often than not, may merely be 
finding a program- -any program (I)--that might possibly 
produce the desired results.) 

One of the functions of the evaluator in the planning phase is 
therefore to gather the kinds of information that may help the adminis- 
trator determine vrtiich courses of action, among several, he should 
pursue. Research literature, expert-practitioners and consultants are 
sources of such information. The evaluator may assist the administra- 
tor in tapping into these sources. 

Another function of the evaluator at this stage is to design an 
evaluation for the selected program. The input-output model offers one 
convenient way of laying out a plan so that it may then be evaluated. 
The plan can then be used to specify what data should be collected and 
how. We can think of that plan as a blueprint with boxes labelled in- 
put, process and output. He can then caatpsre vdmt was intended with 
what actually happened. Thus imagine yourself carrying blueprint I 
from this planning phase along with you into the next two phases of 
evaluation. 



i/Yehezkel Dror, "The Planning Process: A Facet Design," 
Pla nning Programmijig Budgeting. Fremont J. Lyden and Ernest G. 
Miller, eds.., Chicago, Markhaa Publ Co., 1967, p. 99. 
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Implementation Evaluation 



Suppose blueprint I specifies that output is reading scores, that 
process is a reading program, and that inputs are teacliers, students 
^ and reading books. Suppose novr that books don't arrive and that no 
reading is tauglit. It would be misleading to look only at output 
(reading scores) and declare the reading program a failure. For in 
fact the reading program never took place. 

The purpose of iniplementation evaluation is to avoid just such 
faulty reasoning* Since, logically a program can't be evaluated until 
it is installed, implementation evaluation seeks to determine whether or 
not (or to what extent) it has been installed. It atten^ts to determine 
if the inputs (staff, students, materials, facilities) specified in 
blueprint I are actually occurring. (It may also identify what things 
are happening that were not intended.) 

Progress Evaluation 

Orice a program has been installed the major concern becomes the 
extent to viiich it is making progress toward its objectives. Remember 
that according to the rational systems analytic model, the objectives 
constitute the basis for undertaking the program. If this information 
can be provided at appropriate intervals throughout the year it may 
enable the decision-maker to modify the program when objectives are not 
being met. This type of periodic feedback on program performance is 
called 'Normative evaluation^' and is now the commonly accepted mode for 
progress evaluation. 

Outcome Evaluation 

At the end of a program the decision-maker must make some judgment 
concerning tlie worth of the program. The evaluator's task at this point 
is to provide information about program output which can help the 
decision-maker reach that determination. While quantitative data- -in- 
cluding qualitative descriptions and perceptions --can't also be used in 
this assessment of worth. 

Student Exercises 

Students are now ready to use the evaluation simulation of CSE. 
That simulation is available through McGraw Hill. That simulation is 
presently designed for a two--day iittensive seminar to introduce school 
personnel to concepts of evaluation. For purposes of tliis course, 
however, we suggest that the lessons can be spread out over a longer 
period, that students take time to review some of our suggested readings, 
and that an^le time be devoted to discussion of the issues suggested in 
the above Student Reading. 

Suggested Readirigc 

*Klein, Stephen. 'The Center's Chang.ing Evaluation Model." UCLA 
Evaluation Comment , Los Angeles, Center for the Study of 
Evaluation, Vol. 2, No. 4, Jan., 1971. 

ERiC '9.^- 



^Lehman, Heniy. *Th£ Systems Approach to Education. Audiovisual 
jistructi on, Feb., 1966, pp. 144-^148. ^ 



LESSON SDC: 



Comments to the Instructor 

This lesson represents a significant departure from the kinds of 
issues \^Mch emerge as important in most of the literature on evalua- 
tion. The reading for this lesson draws attention to basic issues of 
values and institutional functioning. The lesson is not intended to 
advocate a particular position but rather to bring some controversial 
issues to the attention of the student. We recommend that students read 
all of the suggested readings and that adequate time be set aside for 
discussion of the issues. 

Student Reading 

It is by now a commonplace among historians of education, as well 
as among social and educational critics, that the educational system in 
this country rather faithfully reflects the established structures of 
political a:;d economic power. Education, that 3s, tends to be a follower 
of American society rather than a leader. Some ire now suggesting that 
the same holds true for most educational reseaxcii. What are the aspects 
it reflects? 

During the sixties a breed of scholar-consultants emerged among 
policy-making circles in this country. Regarding themselves as pro- 
fessional reformers,!/ they saw 'Lheir mission as one of upgrading the 
existing bureaucratic system, to make it both more equitable and more 
efficient. Above all they claimed to be nonideological ; that is tc 
say, their role inplied technician, one who identified and solves pro- 
blems within a given instituional framework, while the institutions 
themselves and the values underlying them are assumed and not ques- 
tioned. The involved yet '*non-ideological*^ stance of these scholars 
came to influence and eventually to dominate much of the research that 
was carried out during the decade. Coincidentally, it was in the mid- 
sixties, more precisely with the passing of the BSEA in 1905, that edu- 
cational evaluation suddenly came into its own as a field highly in 
demand. It is not surprising, then, that so much of the literature 
since then seems almost exclusively preoccupied with working out nar- 
row technical problems - how to inprove testing, how to state behavioral 
objectives more Clearly, how to promote program development, how to 



Daniel P. Moynihan, **The Professionalization of Reform," The 
Public Interest, No. 1, Fall, 1965. 
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assess net cost-effectiveness, etc, etc. - all to the conspicuous neglect 
of the more fundamental questions, such as What does school do for child- 
ren anyway? Do we need altogether different arrangements for allowing 
education? 

The most recent development in evaluation to catch on has been the 
systems approach to evaluation, vdiich examines inputs, outputs, and the 
mediating process. It should be noted, however, that all of these ap- 
proaches characteristically confine their attention to description or 
judgment within the parameters of the existing educational system, and 
in its terms. In other words, by implicityly accepting the conventional 
educational framework, evaluators measure progress, success or failure 
against norms which remain immune to question. 

To coincidence between the trend to the scholarly professional re- 
former and the sudden emergence of the educational evaluator as a crea- 
ture in great demand is of great significance for our interests. Of 
special importance is the fact that the 1965 legislation brough evalua- 
tion out of its preferred insulated environment and gave it a direct 
link to public policy-making at the national level, and subsequently, 
through locally planned and aministered programs, at the state and lo- 
cal levels. What is more, the legislative intent of many of the resul- 
ting programs, though couched in educational rhetoric, was social and 
political change. The educational specialists who were called upon to 
evaluate these programs were thus on very unfamiliar ground. It is no 
wonder, then, that their response took the two forms it did: some of 
them reverted to what they'd been doing all along, i.e. honing their 
slrilss in tests and measurements; tlie *'new breed" took a page from the 
book of scholar-consultants, putting themselves in service to adminis- 
trators and decision-makers, and redefining evalioation into the bargain. 

Meanv^iile, by the end of the decade there began to develop a serious 
challenge to the concept of the professional reformer. This came about 
as the result of a growing disillusionment with the ability of educa- 
tional institutions to deliver democratically the services for which 
they were intended. A larger part of the problem, it was determined, 
stemmed from the fact that the professionals who staffed and administered 
these bureaucratic structures had become an interest group in their own 
right, and their natural tendency was to magnify the presumed differ- 
ence in competence between themselves and the lay public in order to 
consolidate and perpetuate their own position of power. As a consequence, 
public services were marked by a lack of democratic control and accoun- 
tability to consumers which in turn produced widespread alienation be- 
tween professionals and clients. In response to this crisis, a new 
kind of reformer arose to champion the rights and interests of the pu- 
blic against the heganony of the professional and technocratic elite. 
What was new about their approach was that it undertook a critique of 
the sti^icture and values of the bureaucratic institutions themselves. 
No longer, they said, must the reformer be content to tinker around with- 
in the system; it is the system itself which needs changing. 

Obviously, one of the prime public institutions to figure in this 
controversy has been the education system. Indeed, schools in areas 
where the breakdown in services has been especially acute - the urban 
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ghettoes - have been major crucibles for the efforts of the new re- 
formers. The pot is kept sijnmering by the related factors of increased 
political consciousness and expectations among the powerless minorities 
and various federal legislation mandating "maximum feasible participa- 
tion" of the poor. In the midst of all this upheaval stands the educa- 
tional evaluator, who lags jsut behind current developments. As des- 
cribed above, he has so far defined his contemporary role narrowly and/ 
or conservatively. 

Some. suggest therefore that the evaluator must now take responsibi- 
lity for finding wrys to integrate developments in social and political 
spheres into his analyses - that he must extend his vision beyond the 
narrow effect (output) of treatRient (process) on students, (input) to 
include questions of basic institutional functioning and values. 

Still a departure from traditional approaches to evaluation, this 
vieivpoint is beginning to emerge with some degree of regularity. As 
you become involved in evaluation, you need to be aware of where you 
stand in this issue. The suggested reading l^^lich follow may help to 
clarify some of these issues for you. Perhaps most important though 
will be the insights you develop over the next years of your practical 
experience in evaluation. 

Suggested Readings 

*Berlak, Harold. "Values, Goals, Public Policy and Educational 
Evaluation," Review of Educational Research , Vol. 40, No. 2 
April, 1970, pp. 261-279. 

*Cohen, David K. "Politics and Research: Evaluation of Social Action 
Programs in Education." Review of Educational Research , Vol. 40, 
No. 2, April, 1970, pp. 213-238. 

*Elboim-Dror, Rachel. "Some Characteristics of the Education Policy 
Formation System," Policy Sciences , 1970. 

*Gla5s, Gene V. "The Growth of Evaluation Methodology," The Evaluation 
Workshop of the Adult Education Research Conference, Minneapolis, 
Minnesota, March, 1970. 

*Myers, Albert E. "The Impact of Evaluative Research on Educational 
PrograJTis for the Poor." Teachers College Record, Vol. 71, No. 3, 
February, 1970, pp. 371-378. 

*Scriven, Michael. "The Methodology of Evaluation." Tyler, Ralph, et. 
al . eds . , Perspectives of Curriculum Evaluation . Chicago , Rand 
Mcnally and Co. , 1967. 

*Stake, Robert E. "Objectives, Priorities, and other Judgment Data." 
Review of Educational Research , Vol. 40, No. 2, April, 1970, 
pp. 181-212. 

^Starred readings are suggested for students. 
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Ai)pendix F 



Teachers College Course Two: 
Qualitative Data its Collection and Analysis: 
An Approach to Problem Slolving 
Detailed Design 



Section I: Field Experiences 

Rationale: In this section a major course-long exercise is sugges- 
ted, as well as nine other shorter exercises. The shorter exercises 
should be carried out sequentially and can be conducted at the same 
tijne that the student conducts the major exercise. The rationale for 
including both a field study and individual exercises related to a 
growing trend or style in the sophistication of the field observer as 
a researcher. 

The growth of students over time related to field work and/or labora- 
tory experiences has been discussed by John Bennett. V His extremely 
useful series of training exercises and presentativon of results ob- 
tained with students over time give us a similar target at which to aijn 
although our process is getting there is somewhat different. In his 
terms, he finds a generalized cultural anthropologist approach emerging 
overtime on the part of his students. A growing awareness on the part 
of the student of the relationships between percept and concept, he 
feels, is a sign of increased methodological sophistication. 2/ 

For Example, the good field worker must work with and through his 
"biases," become aware of them, control them and use them."^/ 

Specifically, students utilizing his experiences begin to evidence 
a general intellectual sophistication as manifested by: (a) a bal- 
anced interest in wholes, parts, details; (b) awareness of the inpor- 
tance of empirical description as well as interpretation; C^) indica- 
tions of an understanding of theory as a body of interpretative prin- 
ciples that mightbe brought to bear cn the data when and if these data 
are sufficiently abundant or suitable for such application of theory. z/ 



—'John W. Bennett, "Individual Perspective in Field Work: An Exper- 
imental Training Coui'se. "Human Organization Research: F 5 eld Relations 
and Techniques , Hd. by Richard Adams and Jack Priess, Homewood, Illinois, 
The Dorsey Press, 1960. . pp. 431-440. 

-/ibid., p. 438. 

l/jbid. , p. 432. 
4/Ibid. p. 438. 
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The reading o£ Mr*. Bennett's training procedures to achieve these 
ends ife hignly recoinmended. These objectives are fundamental to com- 
pleting a qualitatively oriented field stiody and we should hope that 
some of the exercises which follow may approximate his success. How- 
ever, once this core of skills has been acquired a bridge must be es- 
tablished that relates this increased sophistication on the part of the 
student to the actual process of learning to develop new theory, hypotheses 
or ideas, to establish provisional hypotheses or categories and to eval- 
uate the adequacy of these ideas in terms of the relation to the descrip- 
tive systems obtained. It is tJiis end objective which in the writer's 
view justifies includion of a field study as an integral, if optional, 
portion of the course as represented in the basic exercises to follow. 



EXERCISE ONE: Major Exercise 

Objectives: This exercise, ongoing throughout the entire course, provides 
tiie vehicle for integi'ating the use of skills and conepts 
taught through other course materials. This exercise car-^ 
ries the student through the cycle of design, data collection, 
data analysis and reporting in a practical setting. Con- 
tinuity in the study of a single social organization or^ 
educational program, for example, and systematic attention 
to the necessity of integrating the components . of the study 
underscores the issues that are brought forth in the other 
materials , and moves the student jiiore directly into future 
professional application of field work in the evaluative 
process . 

Task Description : 

Students electing to undertake a field study simultaneously with 
the pursuit of their course work might be presented with a menu of 
suggestions for settings or problem-areas that might be applicable to 
their interests. This might include, for example, an "innovative" edu- 
cational program being in^jlemented by a school; a community -mental health 
center; a graduate school; a department in a graduate school; an organi- 
zation responsible for training "paraprofessionals" ; a drug education 
program; a prison setting, etc. 

Initial strategy meetings regarding study design would be held with 
the instructor, or a responsible course assistant. A reading list for 
the entire course should be distributed at that time. The student and 
instructor might discuss which readings might be immediately applicable 
to the student's entrance into the organization in an observer capacity. 
Field relations, client relations, initial participant-observer stance, 
and a general-^ zed notion of what he is looking for might, for example, 
be suggested. Over the course df the year the student would be required 
to keep weekly logs of field activities vSiich would be submitted to 
the instructor for review or individual discussions. The contents of 
this log would indicate such areas as student progress in obtaining 
descriptive information and the selection of his informants, student 
progress in terms of categorization, use of tlieoretical sampling, ana- 
lysis and the generation of working hypotheses. The responsibility of 
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the instructor v/ould be to insure that the student did not tarry too 
long with description alone but moved rather quickly into a framework 
o£ continual analysis o£ the data being generated. 

Supervision would be maintained throu^out the year. Although the 
exercises and readings presented below would ia some cases parallel and 
facilitate the process o£ moving through a field study, the instructor 
must schedule individual time to aid the student ia planning next steps 
and monitoring conceptual difficulties that may arise. 

A final report would be due near the end of the course. Its format 
should parallel in many respects an anticipated professional reporting 
style. That is, the student should include an description of his de- 
sign j data collection, procedures, the results of his analysis, specu- 
lations for future study, and samples of interview schedules or other 
data gathering instruments. Additionally, the student should include 
as an appendix to the report a discussion of the contributions of the 
experience of conducting such a study to his personal and professional 
growth and an indication of potential applications to his career or 
major area of interest. Critical comments would, of course, be encour- 
aged as well. 



EXERCISE TWO 

Objective: The student recognizes that individual differences are a 
significant element in observation and develops increased 
familiarity with his unique patterns of orienting toward 
the world. 

Task Description : 

- Students, organized into teams of three, or four, are asked to take 
notes describing the same five-ten minute segments of behavior of a 
single individual or a series of single individuals. For example, the 
instructor may send students to a busy bus station or other transporta- 
tion centers, there to select persons viho will be observed. Other 
possible settings include toy stores, supermarkets, doctors' offices, 
night-court, a .subway car, etc. Notes taken on the scene are written 
up at home and subsequently distributed in class. 

The class and the instructor analyze similarities and differences 
between various individuals in the groups and between class members from 
various groups. The relevant differences in the descriptions are iden- 
tified and the possible sources of observer differences in reporting 
on the observation would be greater than if the students had been told 
what to look at and instructed as to the concretenss of descriptive 
materials to be obtained. Individual idios^mcracies noted by John 
Bennetty as likely to be revealed by this exercise ,p.re highly use of 



^Ibid., pp. 431-440. 
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adjectives, and the use o£ ready-made frames of reference such as stereo- 
types. Other differences might be noted relating to the drawing or 
failing to draw inferences as to the intent of the actor's behavior, 
the quantity of verbal and non-verbal behavior reported, the extent of 
verbatim match of reported verbal behavior, the extent of attention to 
the sequence of actions, or the extent to which the actor's surroundings 
or his relationship to other actors is reported. 



EXERCISE THREE 

Objective: The student increases his awareness of the selectiveness of 
individioal observer style as the complexity of the obser- 
vational task increases. 

Task Description 

As in exercise two, small teams of three or four individuals are 
given the assignment of selecting and observing the same segments of 
behavior. In this case the targets for observation are groups of indi- 
viduals in spatially bounded environments. Sites for observation might 
include a party, a political or community meeting, a fifteen minute 
segment of a television program, a fountain gathering spot in a park, 
etc. As in the previous exercise, notes taken on the spot are written 
up and distributed to the class. 

The instructor's task is essentially to monitor the student's self- 
evaluation. It Is anticiapted that the students' varying patterns of 
organizing their enviroiiments will be brought into sharper focus as a 
consequence of the increased complexit'' of the task--i.e. the observa- 
tion of a group as opposed to a single individual --and that the notes 
of individiaals within a team will be more widely discrepant than they 
were in exercixe two. John Bennettl/ again gives us some leads as to 
the nature of individual patterns idiich may emerge, particularly a.s 
students move toward a descriptive synthesis of their notes. His obser- 
vations are as follows: 

a) 'The empirical ethnographer 'pattern: concentration upon 

detailed description, recorded systematically and sequen- 
tially« Interpretation, hypotheses, etc., avoided. Gestalts 
rare. 

b) 'The holistic ethnographer' pattern: Concentration upon 
the making of Gestalts of various kinds, especially emo- 
tiona.1 -aesthetic \^^holes; emphasis on color and 'feel' of 
sitTjations. Descriptions show beginnings of interpreta- 
tive construction. 



^ Bennett, op. cit . , p 436. 
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c) 'The social ajithropologist' pattern: Dominant interest 
in structure and system which results in rapid summari- 
zation of detail and its conversion into empirical .des- 
criptions about patterns. Contextual emphasis on 'inter- 
personal relations.!/ 

Other considerations that might arise in the analytical discussions 
might relate to the manrer in which pre-existing theories influence the 
nature of the observations made, to the difficulties of describing 
a complex social situation for purposes of replication, and to problems 
arising from the lack of any general notion on the part of the obser- 
ver as to what to focus upon at the time of entering the settirig. The 
analyses should also include an exploration of what dimensions of ob- 
servation have been left out and why. 



EXERCISE FOUR 

Objective: The student recognizee- that using himself as an instrument 
for making observiitions, as opposed to utilizing techno- 
logical tools, can facilitate as well as distort the pro- 
cess of describing an^l of moving to^vari subsequent under- 
standing. Attention is drawn to his use of sensory mech- 
anisms in processing stimn;li from modalities other than 
the visual and auditory- Data gathering tools are intro- 
duced that promise greater replicabiiity of the recorded 
aspects of verbal and non-verbal behavior, including 
gestures, small and gross movements and facial expressions. 
However, the student comes to recognize that the problem 
o£ categorization and of drawing inferences from the data 
is not completely solved by technology. Replicable data 
are not necessarily objective data. 

Task Description : 

Students are asked to review the readings on specimen records in 
The Midwest and its Children:^ ^ They then select a classroom teacher 
vjho will consent to be observed and to have her verbal behavior simul- 
taneously tape recorded. The student then takes a speciment record of 
about forty-five minutes during i^ich time he also tape records the 
teacher's verbal performance. It would be useful for him to record time 
markers in his running account at no more than five minutes intervals 
to allow for subsequent coordination of the two descriptions. 

Following the observation, the student types his specimen 
record into final fom and gives a copy to the instructor for distribu- 



1/Roger G. Barker and Herbert F. Wright, Midwest and Its Children^ 
The Psychological Ecology of an American Town , Evanston, Illinois: 
Row, Peterson and Co., 1954, pp. 195-224. 
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tion to the entire class. At this point it inay be feasible to do a 
critical analysis of the ^rotocols in a class discussion, attending as 
a group to value judgments v/hich may be noticeable in ±e data and thus 
calling tlie attention of the student -obiverver to biase:. which he has 
not yet systematically identified. Following this session, the stude-it 
would then listen to the entire tape recording of his teacher -observa- 
tion session. This could be done in five minute se'jments, attending 
to the tape while reviewing the specimen record fox the same period. 
Notes would be taken speaking to the issue of the extent and quality of 
information availrble through one recording medium as opposed to another. 
The student should especially attend to and identify the non-verbal, 
verbal and contextual cues which he used in obtaining his specimen re- 
cord. Basically the observer should bejin to orient to the number of 
sensory stimuli to which he attends, and to the inter- relationships be- 
tween the modalities of sight, sound, smell, and kinesthesia which are 
operating to create a total impression. The instructor also might sug- 
gest to the individual analysts before they have begun their analysis 
that they attempt to pinf)oint descriptive statements that refer to the 
intent of the actor (teacher) and examine the validity of these judgments 
in tems of the total context of the observation. Attention would thus 
be drawn to the time sampling implications involving long sequential 
records as opposed to time-sampling or situational -sampling procedures, 
especially as these relate to perceived intent. 

(An optional exercise that would draw out the issue of the actor's 
intent even more clearly would be to write another brief specimen re- 
cord on a teacher and interview her with respect to critical incidents 
in the record, asking her what goals she had been pursuing with these 
actions. Comparisons would then be made between the teacher ^s and the 
observer's perceptions of the situation, alerting the observer to 
potential premature interpretations of intent in the observer's record.) 

Sliould the instructor wish to utilize videotape in order to illustrate 
the advantages and limitations of this medium, he may do so here. Stu- 
dents will recognize the usefulness of data vSiich can be returned to 
again and again for rechecking the validity of categories. But they 
will also see that even this apparently more complete an^''. objective type 
of data can be biased and/ or imccmplete. Spatial cues may be distorted, 
contextual cues may fall outside the camera range, aural cues may be 
masked, etc. Moreover, the available material is still not useful until 
it is condensed and summarized- -introducing problems of anaJ^^^^s not 
dissimilar to those involved in making a specimen record. Viedotape 
may be especially useful in training observers to be sensitive to non- 
verbal cues- -categorization exercises may be begun with the videotape 
sound turned off. (See Exercise Ten) 



EXERCISE FIV E 

Objective: The student gains critical understanding of his personal 
style of interviewing. 
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Task Description : 

In conjimction with his major field study, each student tape records 
at least one and preferably mqre of his interviews. Together he and 
another student, or the instructor, analyze the protocols in tems of 
the four major components suggested by Dohrenwand aiid Richardson.!/ 
These are as follows: 

1. Control of the topics of the interview; 

2. Restriction of the length of the informant's responses; 

3. Restriction of the content of the informant's responses; 

4. Suggestion of the content of the informant's responses. 

Sugsequent practice in tape recording and analysis would be optional 
for each student, but a critique of subsequent progress could be sub- 
mitted as a portion of his year's project. 



EXERCISE SIX 

Objective: The student learns to structure inteview schedules in 

the light of critical hypotheses being developed through 
his field study. 

Task Description : 

Each student is asked to keep- -as part of the weekly log of his field 
study experiences referred to in the major exercise- -a record of the 
types of interview questions he has been posing to his informants. 
At a critical point in each individual's year-long field study, he, or 
he and the instructor together decide that it is time to draw up an 
interview schedule in order to insure the student's systematic gathering 
of the information most crucial to his study and to the report of his 
findings. One or a series of these structured interviews may be drawn 
up by the student and submitted to the instructor for review and con- 
structive criticism. The interview schedule (s) then become appended to 
the student's evaluation study along with his critical comments as to 
the usefulness of the instrument (s) in sharpening his conceptual focus 
and clarifying what it is he is looking for. 



— Barbara S. Dohrenwend and Stephen A. Richardson, "Analysis of 
the Interviewer's Behavior," Human Organization, Vol. 15, No. 2, 
pp. 29-32. 
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EXERCISE SEVEN 



Objective: The student becomes aware o£ the effects of roles he plays 
along the participant-observer dimension with respect to 
qualitative differences in the data he is able to collect. 
He receives practical experience in the sometimes con- 
flicting demands of personal involvement and measured 
detachment in data collection. 

Task Description : 

Each student is to select two of the following three role positions 
from \^^ich to conduct two two-week studies. He may choose the role of 
a detached observer, the role of a participant who chooses to join an 
organization for the ovservational purposes of his study, as in "When 
Prophecy Fails"!/ or the role of an involved participant. Preliminary 
readings and/or orientation to impli cut ions of these various roles should 
be plaimed prior to the initiation of the studies. A discussion of 
field relations and problems of gaining access to the population studied 
should be scheduled. An example of a study utilizing the participant 
extreme in terms of level of involvemer.t might revolve around current 
role functions for the individual- -hi? job and work environment, his 
functions in community settings ? his relationships to his neighbors in 
the social structure of his apartmeiit building, existing organizational 
relationships, etc. The parallel otudy to be conducted simultaneously 
might well be selected to coincide with the area he selected for func- 
tioning as a participant. If for example the individual is functioning 
as a teacher he might want to do his observations in other classrooms 
in geographically different schools. The instructor should schedule 
time with students individually to h^ip them define two-week areas of 
study. 

A brief report would be expected at the end of three weeks which 
would define the general nature of the design., data collection techni- 
ques, nature of data collected, preliminary findings for the ti^jo studies, 
and a detailed elaboration of the advantages and limitations of con- 
trasting role stances as they effect the quality of the data collected 
and the ability to remove oneself sufficiently from identification with 
participants to generate new hypotheses or ?iigher order generalizations. 
The student might direct some attention to describing the changes in 
his role over time and the effects of this on the collection of particu- 
lar information • Individual preferences for one cf the extreme roles 
as opposed to another might become items for discussion with respect 
to the particular nature of data needed for studies in different stages 
of development ♦ Particular emphasis might be placed on finding ways 
to flexibly shift role stances depending on study need. 



-Henry W. Riecken, ''The Unidentified Interviewer,*' Issues in 
Participant Observation , Ed. by McCall and Simmons, Reading, Mass., 
Addison Wesley, 1969, pp. 39-44. 
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EXERCISE EIGHT 



Objective; Student learns that categories must be well-defined before 
replication of results can be achieved. 

Task Description : 

The instructor assigns to groups of three or more observers several 
generally defined topics for observation. If videotapes of films 
utilizing other than classroom seetings are availab,e use these, since 
the more unfamiliar the material the better. 

This time observation is not blind. Rather, the instructo-vassigns 
a series of loosely defined categories as a structure for observ^ation. 
Students may, for example, be told to look for communication patterns, 
interaction patterns, decision-making patterns, etc. Students then 
write up their notes around the assigned topics; they are duplicated 
and shared with the class. As in the previous exercises, the instructor 
then monitors and guides the students' critical analyses of similarties 
and differences in the protocols, and their speculations as to why these 
occur. Particular enphasis should be drawn to the need to utilize 
categories for observation which have meanings held in comnon by the 
observation team; the greater the level of inference demanded from 
the observer, the less potentially replicable are his findings. 

EXERCISE NINE 

Objective : Students begin to recognize the advantages of theoretically- 
based categories as they move from the descriptive level 
through the synthesis of y^iat they Yiave observed, to the 
detection of meaningful patterns. IVhile instrumentation 
may in some instances aid this process, in the case of 
categories not theoretically based it may actually impede 
analysis. 

Task Description : 

A videotape of classroom interaction is shown to the students 
without accompanying sound in order to draw their attention to non-verbal 
dimensions to be observed. Students are then shown the videotape a 
second time with the sound turned on and asked to take notes which will 
enable them to describe what they have seen. The descriptions are then 
read alound and the participants mutually point out various dimensions 
to which other stiodents were attending or not attending as the case 
may be. The group attempts to generate ideas regarding the theoretical 
relevance of what they have seen as opposed to merely identifying minute 
details which suggest nothing more than a "so what?" response. Once 
again the instructor emphasizes the need for an interplay between data 
and theory and encourages the class to find ways to order their observa- 
tions into patterns or generalized statements about what has occurred. 

Two or three observation instruments are then distributed for 
student review at home during the week. Examples might be the Flanders 
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Systean of Interaction Analysis, Bruce Joyce's!/ recently revised version 
of the instrument at Teachers College by Felice Cordis entitled "Settings 
for Instruction: Classroom Structure," During the next class session 
the students try out these instruments on the same section of videotape 
shown on the previous week. It is to be assumed that the discussion 
following the tape would bring out the advantages of utilizing pre- 
determined categories in systematizing data, while simultaneously ue-- 
monstrating that such cate^ries may make greater demands for abstract 
(and therefore more idiosyncratic) judgments on the part of the observer. 
It should become evident that on the one hand categories must be well- 
defined if they are to be replicable, and that on the othcm hand if 
the thebry is not sufficiently comprehensive, important dimensions of 
perceivable descriptive data will of necessity be excluded from categori- 
zation. The exercise should raise questions about the merit of various 
instruments in generating useful theoretical statements about observed 
events. 

EXERCISE TEN 

Objectives: The student becomes aware of the usefulness of speculating 
in advance about the general nature of what he looking for-- 
i.e. a tentative hypothesis. Simultaneously he retains enough 
flexibility to mdify his original ideas on the basis of his 
analysis of empirical data, and using that data he develops 
new categories or ways of observing that need to be explored. 

Task Description: 

Each student is asked to think about children he has observed in 
school and generate some tentatix^e h)TX)theses as to what aspects of 
their current behavior may have beei affected by their watching of TV. 
Specifically, what is TV teachin^-^ children, or not teaching children? 
Based on this speculative framework, the student should generate notions 
as to what he expects to see on the segment of IV aimed specifically 
at children. Preliminary categoreis are formulated. The student then 
observes one four-hour Saturday mominf segment of childrens television, 
and makes preliminary descriptive categorizations of the data. 

The student then compares his obtained data with his pre-existing 
ideas and categories. His next step is to suggest in a brief paper for 
his instructor the hypotheses which he has developed as a result of 
systematic observation and categorization and to propose ways in which 
to further insights might be developed by further observation of children. 
Papers could be distributed to the class in order to illustrate the 
wide range of theoretical stances which different observers might take 
in approaching similar data. ER5)hasis on a single-factor analysis shoul 
be avoided _in. favor of a consideration of patterns. 



Section II - Issues Outline 

Rationale: Following is one possible functional organization of. the 
course. It has been designed to coincide with readings indicated in 
Section III - Bibliography. 
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Part I . Background Material^ Regarding Field Studies and Participant 
Observation 

A, History and definition of field study approaches 

B, Functions of qualitative analysis 

C, Limitations and advantages of participant observation 
studies 

Part II, Background Materials Regarding, the Philosophy of Science 

A, Introduction to issues pertinent to the collection and 
analysis of qualitative data 

B, Explication of the position that no science is value-free. 
Need for the recognition and explication of personal 

bias in the search toward objectivity 

C. Replication of research as an objective to be pursued 
Part III, Learning How to Observe 

A, The observer's role - general comments 

1, Part icipant-as -observer vs, observer- as -participant • 
Stances and implications for nature of data collected 

2, The balance between personal involvement and objec- 
tive detachment 

3, Ihe establishment and maintenance of role relation- 
ships in the field 

4, Role shifts over time and implications for the nature 
of data collected 

B, Strategies for Interviewing - Some Approaches 

. C , Some Techniques and Tools for Recording Qualitative 
Data 

D. Observation Strategies 

1, An overview of current approaches 

2, Deciding when and where observations will take place 

3, Deciding what units will be observed 

a. Units of analysis 

• ,/ 

b. Selectivity of observation through the role of 
history and the interplay of theory and data 
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Part IV. Learning to Describe 

A. Definition o£ categories 

B. Categories as representations of means of structuring 
reality. Relationship to language and thought 

C. Possible sources for categorization 

D. Means of generating new categories. Constant comparative 
method of analysis. Theoretical sampling. Insight 
stimulating cases 

Part V . Instrumentation 

SajTiples of refinement of categories through development 
of instrumentation. Examples; of theory-based observation 
instruments and ''theory-generating*^ instruments 

Part VI, Obtaining Higher-order Generalizations from Qualitative Data 

Part VII. Analysis of Qualitative Data 

A. Deductive and inductive approaches - geneial comments 

B. Generation of hypotheses; generation of theory 

C. 'Lvo techniques of qualitative analysis 

D. Maintaining quality of data: validity 

E. Evaluation of hypotheses and credibilit>'oof theory 

F. Cross references to case studies. Addi tonal case studies 
Part VIII. Final Reporting 

A. Issues relating to ethics and politics 

B. Reporting format 

Part IX. Related Methodological Approaches to Qualitative Data 
Collection and Analysis 

A. Survey techniques and qualitative analysis 

B. i^^si- Experimental research designs 

C. Ecological and ''classical^* issues in research 

D. Ob<= ,^-rvation and psychological tests 

E. Example of an analysis of complex organizations 
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Section III - Bibliography 

The bibliography has been organized to parallel or coincide with 
Section II. Ihelatter is intended to illustrate in a logical sequence 
a few of the fundamental issues we would touch upon if teaching the 
course. Readings considered particularly useful have been starred. 

Part I: Background materials regarding field studies and participant- 
observation. 

A. Histor/ and definition of field study approaches. 

Becker, Howard S. , Blanche Geer, Everett C. Hughes, Anselm Strauss. 
Boys in Miite . Chicago, University of Chicago Press, 1961. 
Chapter 2. 

Katz, Daniel. "Field Studies." Research Methods in the Behavioral 
Science. Ed. by L. Festinger and D. Katz. New York, The Dryden 
Press, 1953. pp. 56-97. 

**McCall, George J. and J. L. Simmons, eds. Issues in Participant 
Observation. A Text and Reader. Reading, Mass., Addison Wesley 
Publishing Company, 1969. 

*Powdermaker , Hortense. "Fieldwork." International Encyclopedia of 
the Social Sciences. New York, The Macmillian Co. , 1968, Vol. 5, 
pp. 418-424. 

Strauss, Anselm, et. al . "The Process of Field Work." Issues in 
Participant Observation. Ed. by G. J. McCall and J.L. Simmons. 
Reading, Mass., Addison Wesley Publishing Company, 1969. pp. 24-27. 

Strauss, Anselm, et. al . Psychiatric Ideologies and Institutions. 
New York, The Free Press, 1964. pp. 18-37. 

*Whyte, William F. "Observational Field-Work Methods." Research 

M ethods in Social Relations. Ed. by M. Jahoda, et. al. New York, 
The Dryden Press, 1951. pp7 495-513. 

Valentine, Charles, "Appendix: Toward an Ethnographic Research 
Design." Culture and Poverty . Chicago, University of Chicago 
Press, 1968. 

B. Functions cf qualitative analysis. 

*Barton, Allen H. and Paul F. Lazarsfeld. "Some Functions of Qualitative 
Analysis in Social Research." Issues in Participant Observation . 
Ed» by McCall and Siinmons. Reading, Mass., Addison Wesley 
Publishing Company, 1969. pp. 163-195. 

Strauss, Anselm L. "Fieldwork, Theoretical Sampling and Educational 
Research." San Francisco, University of California (unpublished) 



**Major source for this bibliography. Suggest it be read in entirety 
and tehn re-read specifically. 
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C. Issues and debates concerning limitations and advantages o£ 
participant -observation studies. 

^Becker, Howard S. and Blanche Geer. "Participant Observation and 
Interx'-iewing: A Comparison/' Issues in Participant Ob servation . 
Ed. by McCall and Simmons. Reading, Mass., Addison Wesley 
Publishing Coinpany, 1969. pp. 322-331. 

Becker, Howard S. and Blanche Geer. "Rejoinder/' Issues in Participant 
Observation . Ed. by McCall and Simmons. Reading, Mass., Addison 
Wesley Publishing Coinpany, 1969. pp. 338-341. (refers to Trow 
article) 

Bierstadt, Robert. "A Critique of Empiricism in Sociology. American 
Sociological Review , 14:584-592, 1949. 

*Dean, John P. , Robert L. Eichhom, and Lois R. Dean. "Limitations 
and Advantages of Unstructured Methods.'' Issues in Participant 
Observation . Ed. by McCall and Simmons, Reading, Mass. , 
Addison Wesley Publishing Company, 1969. pp. 19-23. 

Trow, Martin. ''Comment.'' Issues in Participant Observation . Ed. by 
McCall and Simmons. Reading, Mass., Addison Wesley Publishing 
Company, 1969. pp. 332-337 (refers to Becker and Geer article) 

*Zelditch, Morris, Jr. "Some Methodological Problems of Field Studies." 
Issues in Participant Observation . Ed. by McCall and Simmons 
Reading, Mass. , Addison Wesley Publishing Coirpany, 1969. pp. 5-18. 

Part II: Background materials regarding the philosophy of science. 

A. Introduction to issues pertinent to the collection and analysis 
of qualitative data. 

*Kaplan, Abraham. The Conduct of Inquiry. Methodology for Behavioral 
Science. San Francisco, Chandler Publishing Co., 1964. 
pp. 3-83, 198-214, 370-397. 

B. Explication of the position that no science is value free. 
Need for recognition and explication of personal biases in 
the search toward objectivity. 

*Bennett, John W. , '^Individual Perspective in Field Work: An 
Experimental Training Course." Hunian Organization Research : 
Field Relations and Technicpies , Ed, by Richard Adams ^ Jack 
Preiss. Homewood, Illinois, The Dorsey Press, 1960. pp. 431-440. 

*De Laguna, Frederica. ''Some Problems o£ Objectivity in Ethnology," 
Man, December, 1957. pp. 179-182. 

*Gouldner, Alvln W. ''Explorations in Applied Social Science," 
Social Problems , 3:169-181, No. 3, 1956. 



-114- 



^Kaplan, Abraham. The Conduct of Inquiry. Methodology for Behavioral 
Science . San Francisco, Chandler Publishing Co„, 1964. ~ 
"pp. 3g"8-410; 126-144. 

*Merton, Robert K. ''Selected Problems of piela Work in the Planned 
Community/' Anerican Sociological Review , 15: 304-312, 1947. 

*Gunnar, Myrdal. Objectivity in Social Research . Pantheon Books, 
New York, 1969. 

Robers, David. 110 Livingston Street; Politics and Bureacracy in 
the New York" City Schools . New York, Random House Press/ 1968. 

*Scriven, Michael. Indiana University. 'Value Claims in the Social 
Sciences . " Publication #125 of the Social Science Education 
Consortium , Irving Morrisett, Executive Director, University of 
Colorado, Boulder, Colorado, pp. 1-39* 

Znaniecki, Florian. "Human Actions" (Chapter 7). Cultural Sciences , 
Their Origin and Development . Urbana, The University of Illinois, 
1952. pp. 186-193. 

Znaniecki, Florian. "Chapter IV; the Practical Organization of ' 
Reality," Cultural ! ities , Chicago: University of Chicago 
Press, 1919, 145-154. ' ~ 

C. Replication of research as an objective to be pursued. 

*"Field Methods and Techniques, Observr^tion - V." " Human Organization .' 
Vol. 10, No. 3, Fall 1951. pp. 39-4U. 

*Lewis, Oscar. "Controls and Experiments in Field Work." Anthropology 
Today. Ed. by A.L. Kroeber. Chicago, University of Chicago 
Press, 1953. pp. 452-475. 

Part III: Learning how to observe. 

A. The observer's role- -general comments. 

* Junker, Buford, H. Pielch^ork> An Introduction to the S qcial Sciences. 
Chicago, University of Chicago i>ress, 1560. pp. 32-157. 

Olesen, Virginia L. and Elvi Waik Whittaker. 'Tlcle-Making in 
Participant Observation: Processes in the Research -Actor 
Relationship." Human Organization, 26:273-281, No. 4, 1967. 

1. Participant -as-observer vs. observer-as -participant. Stances 
and iinplicationis for the nature of data collected. 

Babchuk, Nicholas. "The Role of the Researcher as Participa:tt 
Observer and Participant -as-Observer in the Field Situation." 
Human Organization , 14:225-228, 1962. 
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Gold, Raymond L- "Roles in Sociological Field Observaticrs/' 

Issues in Participant Observation . Ed. by McCall and Simmons. 
Reading, Mass., Addison Wesley Publishing Ccmpaxx/, 1969. pp. 30-38. 

*Riecken, Henry W. ''The Unidentified Interviewer." Issues in 
Participant Observation. Ed. by McCall. and Simmons. Reading, 
Mass. , Adclison Wesley Publishing Company, 1969. pp. 39-44. 

*Schwartz, Morris S. and Charlotte G. Schwartz. "Problems in 

Participant Observation." Issues in Participant Observation . 
Ed. by McCall and Sijnmons. Reading, Mass., Addison Wesley 
Publishing Company, 1969. pp. 89-104. 

*Vidich, Arthur J. "Participant Obse::'^/ation and the Collection and 
Interpretation o£ Data." Issues in Parti "zipant Observation . 
Ed. by McCall and Simmons. Readnjig, Mass. , Addison Wesley 
Publishing Company, 1969. pp. 78-86. 

2. The balance between personal involvement and objective detachment. 

McCall, George J. and J. L.. Simmons, eds- Issues in Participant 
Observati on. Reading, Mass., Addison Wesley Publishing Co. , 
19691, Chapter 4. pp. 77. 

Miller, S. M. "The Participant Observer and 'Over-Rapport. '" 

Issues in Participant Observation . Ed. by McCall and Simmons. 
Reading, Mass. , Addison Wesley Publishing Company, 1969. pp. 87-88. 

Powdermaker, Hor tense. Stranger and Friend. The Way of an Anthropologist . 
New York, W.W. Norton and Company, 1966. 

*Wax, Rosalie Hankey. "Field Methods and Techniques: Reciprocity 
as a Field Teclinique.^' Human Organization , 11:34-57, No. 3, 
1952. 

3. The establishment and maintenance o£ role relatioi:ships in 
the field. 

*Argyris, Chris. "Creating Effective Relationships in Organizations." 
Humari Organization Research. Ed-, by R.N. Adams and J.J. Preiss. 
Homewood^ 111., Dorsey Press, 1960. pp. 109-123, 

Dean, Joha P. , Robert L. Eichhom, and Lois R. Dean. "Establishing- 
Field Relations." Issues in Participant Observation. Ed. by 
McCall and Simmons* Reading, Mass. , Addison Wesley Publishing 
Company, 1969. pp. 68-69. 

Kahn, Robert and Floyd Mann. "Developing Research Partnerships." 
Issues in Participgoit Observation . Ed. by McCall and Sijnmons 
Reading, Mass., Addison Wesley Publishing Company, 1969. pp. 4S-51 

*McCall, George J. and J.L- Simmons, eds. Issues in Participant 

Observation . Reading, Mass., Addison Wesley Publishing Company, 
1969, Chapter 2. pp. 28-29. 
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^Rodman, Hyman and Ralph Kolodny. '^Organizational Strains in the 
Researcher-Practitioner Relationship." HumaJi Organization , 
23:171-182, 1964* 

*Scott, W. Richard* 'Tield Work in a Fomtil Organization: Some 
Dilemmas in the Role o£ Observer/' H uiian Organization , 
22:162-168, 1963-1964. 

4. Role shifts over tijne and implications for the nature of the 
data collected. 

*Janes , Robert W. ^'A Note on Phases of the Community Role of the 
Participant Observer." Issues in Participant Observation . 
Ed, by McCall and Siinmons^ Reading, Mass. , Addison Wesley 
Publishing Co., 1969. pp. 52-60. 

S;trauss, Anselr.i, et, a l. ''Field Tactics.'' Issues in Participant 
Observation . Ed. ¥y McCall and Siinmons, Re ading , Mass.jv 
Addison Wesley Publishing Co., 1969. pp. 70-72. 

D. Observation strategies • 

1. An overview of current approaches. 

Kaplan, Abraham. The Conduct of Inquiry. Methodology for Behavioral 
Science . San Francisco, Chandler Publishing Co., 1964. 
pp. 126-144. 

'^McCall, George J. and J. L. Siinmons, eds. Issues in Participant 
Obsei-vation . Reading, Nlass., Addison Wesley Publishing Co., 
1969. Chapter 3. pp. 61-67. 

*Weick, Karl E. ''Systematic Observational Methods.'' Handbook of 
Soc ial Psychology . Ed. by Gardner Linzey. Reading, Mass., 
A^aison Wesley Publishing Co., 1954, pp. 357-451. 

2. Deciding vAien and v/here observations will take place. 

*Field Methods .and Techniques, Observation- -I I." Human Organization . 
9:28-29, Fall, 1950. 

*Field Methods and Techniques , Observation- -I I.'* Human Organization . 
9:30-31, Winter, 1950. 

3. Deciding what units will be observed. 

a. Units of analysis 

Barker, Roger G. "E:cplorations in Ecological Psychology." 
American Psychologist . 20:985-988, 1965. 

*'Tield Methods and Techniques, Observation III." Hum>an Organl^v^tion. 
10:35-36. Spring, 1951. 
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*''Field Methods and Techniques, Observation--IV/' Human Organization . 
10i33-34, Summer, 1951. 

b. Selecti.vity of observation through the role of hisory and 
the interplay of data and theory. 

^Barker, Roger G. and Paul V. GuTip. Big School, Small School . 
Stanford, Stanford Ifriiversity Press, 1964. 

^Barker, Roger G, Ecological Psychology. Concepts and Methods 
for Studying the Environment o f Human^ehavior. Stanford, 
Stanford IMiversity Press, 1968. 

Barker, Roger G. , Ed. The Stream of Behavior. Exploration of its 
Structure and Cont :entT New York, Apple ton- Century Crofts, 
1963. 

* Junker, Buford E. Fieldvvork: An Introduction to the Social 

Sciences . Chicago, University of Chicago Press, 1960. pp. 12-31. 

*Kaberry, Phyllis, "Malinowski' s Contribution to Field Work 
Methods and the Writing of Ethnography.'' Man And Culture . 
Ed. by lUymond Firth. London, Rout ledge and Kegan Pal, 
1957. pp. -1-91. 

Lynd, Robert S. and Helen Merrell Lynd. Middletom . New York, 
Harcourt Brace and Company, 1P29. 

Piaget, Jean. The Origins of Intelligence in Children . New York 
International Universities Press, Inc., 1952. pp. 359-376. 

*Smith, Louis M. and William Geoff ry. The Conylexities of an Urban 
C las sr oom . An Analysis Toward a General Theory of Teaching^ 
New York, Holt, Rinehart and Winston, Inc., 1968. 

Part IV. Learning to Describe 

A. Definition of categories. 

^Becker, Howard S. and Blanche Geer. ^'Participant Observation: 
The Analysis of Qualitative Field Data." Human Organization 
Research . Ed. by R. N. Adams and J. J. Prer.ss* Homewood, 
111., Dorsey Press, 1960. pp. 267-289. 

(Guest, Robert H. ''Categories of Events in Field Observations." 

Human Organization Research . Ed. by R. N. Adams and J. J. Preiss. 
Homewood, 111., Bjrsey Press, 1960, pp. 225-239. 

Wolff, Kurt H, "The Collection and Organization c-^ Field Materials: 
A Research Report," Human Organization Research . Ed. By 
R. N. Adams and J. J. Preiss, Homewood, 111., Dorsey Press, 
1960. pp. 240-254. 
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B. Categories as representatives of means of structuring reality: 
relationship to language and thought. 

*Brown, Roger and Eric A. Lenneberg. "A Study in Language and 
Cognition/' Journal of Abnormal and Social Psychology , 
49:454-462, 1954. 

Conklin, Harold C. '^Hanunoo Color Categories." Southwestern 
Journal of Anthropology, 2:339-344, 1955. 

C. Possible sources of categories 

Berelson, Bernard. Content .^j^ialysis in Communication Research . 
Glencoe, 111. , The Free Press, 1952. pp. 114-168, 

*Glaser, Barney G. ajid Anseljn L. Strauss. The Discovery of Grounded 
llieory. Strategies for Qualitative Research . Chicago, Aldine, 
1967. Chapter VII. 

Guest, Robert H. "Categories of Events in Field Observations." 

Human Organization Research . Ed. by R.N. Adams and J.J. Preiss. 

Heyns, Roger W. and Alvin F. Zander. "Observation of Group Behavior." 
Research Methods in the Behavioral Sciences . Ed. by Leon 
Festi iger and Daniel Katz. New York, Holt, Rinehart and 
Winstcn, 1953. pp. 381-417. 

Whyte, William F. Men at Work . Homewood, 111. , Dorsey Press, Inc. 
and Richard UTTnTinTTnc. 1961. Chapter 2. 

White, WilliajTi P. Psychosocial Principles Applied to Classroom 
Teaching . New York, McGraw-Hill, Inc. 1969. 

D. Means of generating new categories. Constant comparative 
method of analysis. Theoretical sampling. Insight 
stimulating cases. 

*Dean, J. P. '^The Analysis of Insight-Stimulating Examples.*' 

Research Methods in Social Relations by Claire Sllitz, Marie 
Jahoda, Morton Deutsch, Stuart W. Cook. New York, Henry Holt 
and Co., 1959. pp. 59-65* 

*Glaser, Barney and Anselm L. Strauss. The Discovery of Grounded 

Theory: Strategies for Qualitative Research . Chicago, Aldine 
Publishing Co. , 196T Chapters III and V. 

* Jackson, Philip W. Life in Classrooms . New York, Holt, Rinehart and 
Wini>ton, Inc., 1968. • 

E. Levels of inferences involved in descriptive data and 
descriptive synthesis or summari zat i on. 
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Barton, Allen and Paul Lazarsfeld, "Some Functions of Qualitative 
/nalysis in Social Research*" Issues in Participant Observation > 
Publishing Company, 1969 • pp* 163-195. 

*Kluckhom, Clyde • "The Study o£ Culture/' The Policy Sciences , 
Ed. by Daniel Lemer and H.D. Lasswell. Stanford, Stanford 
University Press, 1951. pp- 86-101. 

Part V. Instrumentation 

Samples of refinement of categories through development of 
instrumentation. Exanples of theory-based observation in- 
strmients and "theory-generating" instruments. 

Biddle, Bruce J.. 'Wethod and Concepts in Classroom Research." 
Revie w of Educational Research , 37 : 337-357 , 1967 . 

*Gujnp, Paul V. "The Classroom Behavior Setting: Its Nature and 

Relation to Student Behavior*" Final Report, U. S. Office of 
Education, Midwest Psychological Field Station, University 
of Kansas, July, 1967. 

Joyce, Bruce R* and Berj Harootunian. The Structure of Teaching. 
Chicago, Science Research Associates, 1967. 

*Simon, Anita and E. Gil Broyer. Mirrors for Behavior, Ar. .Anthology 
of ClassTv . om Observation Instruments . Philc delphia. Research 
for Bette> Schools, 1967. 

*Simon, Anita and E. Gil Broyer, eds. Mirrors for Behavior, Summary . 
Philadelphia, Research for Better Schools, 1970. 

Part VI. Obtaining Higher-order Generalizations from Qualitative 
Datf^. Explanation. Descriptive vs. Propositions. 

*Glaser, Barney G. And Anselm L. Strauss. The Discovery of Grounded 
Theory. Strategies of Qualitative Research . Chicago, Aldine 
Press, 1967. Chapter X. 

*Kaplan, Abraham. The Conduct of Inquiry. Methodology for Behavioral 
Science . San Francisco, Chandler Publishing Co., 1964. 
pp. 327-369. 

*"Causal Propositions." Issues in Participant Observation. Ed. by 
McCall and Simmons. Reading, Mass., Addison Wesley Publishing 
Co. , 1969. pp. 254-257. 

*McCall, George J. "The Problem of Indicators in Participant Obser- 
vation." Issues in Participant Observation ^ Ed. by McCall 
and Simmons. Reading, Mass. , Addison Wesley Pub. Co. , 1969. 
pp. 230-239. 
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*Polya, George. Mathematics and Plausible Reasoning, II. Patterns 
of Plausible Inference . Princeton, J.J., Princeton Lfniversity 
Press, 1954. 

*Sellitz, Claire, Marie Jahoda, Morton Deutsch, and Stuart W. Cook. 
Research Methods in Social Relations. New York, Henry Holt 
and Co. , 1959. Chapters 3 and 4. 

*Ananiecki, Floria. "The Historical Evolution o£ Generalizations." 
C ultural Sciences: Their Origin and Development . Urbana, 
111., University of Illinois Press, 1952. ppl ^151-165. 

Part VII. Analysis o£ Qualitative Data 

A. Deductive and inductive approaches; general comments. 

Znaniecki, Florian. "Qiapter V: Criticism of Some Methodological 
Tendencies." Methodology of Sociology . New York, Farrar 
and Rinehart, Inc., 1934. 

*Znaiiiecki, Florian. "Methodological Trends in Sociological Research." 
Sociology an^.. Social Research , 33:10-14, 1948. 

B. Generation of hypotheses; generation of theory. 

Barton, Allen H. and Paul F. Lazarsfeld. "Qualitative Support of 
Theory." Issues in Participant Observation . Ed. by McCall 
and Simmons^ Reading, Mass. , Addison Wesley Pub, Co., 
1969, pp. 239-244. 

*Dalton, Melville. "Preconceptions and Methods in Men Who Manage." 
So ciologists at Work . Ed» by Philip Hammond. Garden City, 
N.f. , Anchor Books, 1967. pp. 59-1] 0. 

*Dean, John P., Robert L. Eichhom, and Lois R. Dean. "Fruitful 

Informants for Intensive Interviewing." Issues in Participant 
Observation^ Ed. by McCall and Simmons. Reading, Mass., 
Addison Wesley Pub. Co., 1969. pp. 142-143. 

*Geer, Blanche. "First Days in the Field: A Chronicle of P.esearch 

in Progress." Issues In Participant Observation . Ed. by McCall 
and Simmons. Reading, Mass., Aildison Wesley Pub. Co., 1969 
149-162. 

''•Glaser, Barney G. and Anselm L. Strauss. The Discovery of Grounded 
Theory^ Strategies for Qualitative Research . Chicago, Aldine 
Press, 1967. Chapters I, II, IV, VI, VIII, and XI. 

*Glaser, Barney G. and Anselm L. Strauss. The Discovery of Grounded 
Theory. Strategies for Qualitatiye Research . Chicago, Aldine 
Press, 1967. Chapter III, pp. 45-77. 
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Kaplan, Abraham. The Conduct of Inquiry, Methodology for Behavioral 
Science, San Francisco, Chandler Pub. Co*, 1964. pp. 294-326 



*McCall an.d Simmons, eds. Issues in Participant Observation . 

Reading, Mass., Addison Wesley Pub. Co., 1969. Chapter V, pp. 142. 

C. Two Techniques of qualitative analysis 

*Glaser, Barney G. "The Constant Comparative Method of Qualitative 
Analysis." Issues In Participant Observation . Ed. by McCall 
and Sinimons. Reading, Mass Addison Wesley Pub. Co., 1969. 
pp. 216-228. 

*Robinson, W,S. "The Logical Structure of /nalytic Induction." 

Issues in Participant Observation . Ed. by McCall and Simmons, 
Reading, Mass., Addison Wesley Pub. Co« , 1969. pp. 196-204* 

^"Sampling." Issues in Participant Observation , Ed. by McCall and 
Sirnmons, Reading, Mass., Addison Wesl'^y Pub. Co., pp. 64-67. 

^Turner, Ralph H. "The Quest for Universals in Social Research." 
Issues in Participant Observation . Ed. by McCall and Simmons, 
Reading, Mass. , Addison Wesley Pub. '".o., 1969. pp. 205-215. 

D. Maintaining the quality of the data: validity 

*Dean, John P. and William F. Whyte. "How Do You Know If the Informant 
is Telling the Truth? Issues in Participant Observation . Ed. 
by McCall and Simmons. Reading, Mass., Addison Wesley Pub. Co., 
1969. pp. 105-114. 

*McCall, George J. "Data Quality Control iu Participant Observation." 
Issues in Participant Observation . Ed. by McCall and Simmons. 
Reading, Mass. , Addison Wesley Pub. Co., 1969. pp. 128-141. 

*Vidich, A. And J. Bensmen. "The Validity of Field Data." Human 
Organization . 13:20-27, Spring, 1954 

E. Evaluation of hypotheses and credibility of theory. 

*Becker, Howard S. "Problems of Inference and Proof in Participant 
Observation." Issues in Participant Observation . Ed. by McCall 
and Simmons. Reading, Mass., Addison Wesley Pub. Co., 1969 • 
pp. 245-234. 

*Glaser, Barney and Anselm L. Strauss. The Discovery of Grounded 
Theoiy. Strategies for Qualitative Reseafdv i Chicago, Aldine 
Press, 1967. Chapter IX, 

*McCall and Simmons, eds. Issues in Participant Observation. Reading, 
Mass., Addison Wesley Pub. Co., 1969. Chapter VI, pp. 229-230. 

F. Cross references to case studies. Additional case studies. 
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Becker, Howard S. Outsiders > Studies in the Sociology of Deviance , 
London, Collier-Macmillan Ltd, , 1963, 

Blau, Peter M, "Co-operation and Competition in a Bureaucracy/' 
American Journal of Sociology , 59:530-535, 1947/1948, 

Evans -Pritchard • Witchcraft, Oracles, and Magic Among the Azande , 
Oxford, England, Clarendon Press, 1967, 

G^ffman, Irving. Asylums , Chicago, Aldine Pub, Co,, 1962, 

Henry, Jules, "Docility or Giving Teacher What She Wants," Journal 
of Social Issues , 11:33-41, 1955, 

Malinowski, Bronislaw, Argonauts of the Western Pacific, New York, 
E,^, Dutton and Co,, Inc, , 1961, 

In additon to the readings listed above it is suggested that the 
student explore case studies listed elseqhere in this bibliography. 
For example, several qualitative studies dealing primarily with 
educational issues are listed below for purposes of cross-reference. 

Barker, Roger G, and Paul V, Gump, Big School -Smcill School , 
Stanford, Stanford University Press, 1964, 

Jackson, Philip W, Life in Classrooms , New York, Holt, Rinehart 
and Winston, Inc, 1968, 

Rogers, David, 110 Livingston Street: Politics and Bureaucracy m 
the New York Citv Schools, New York, Random House Press, 
1968. 

Smith, Louis M, and William Geoffry. The Complexities of an Urban 
Classroom. Nr York, Holt, Rinehart and Winston, Inc, , 1968, 

B. Strategies for interviewing - some approaches. 

*Dohrenwend, Barbara S, and Stephen A. Richardson. "Analysis of 
the Interviewer's Behavior," Human Organization , Vol, 15, 
No, 2. pp. 29-32. 

Gross, Neal and Ward S. Mason. "Some Methodological Problems of 
Eight-Hour Interviews." Abstract, The American Journal of 
Sociology . Vol. LIV, No. 3, November 1953. pp. 197-204. 

*Merton, Robert K. and Patricia L. Kendall. 'The Focused Interview." 
The Language of Social Resea rch, Ed. by Paul F. Lazarsfeld and 
Morris Rosenberg. New York: The Free Press, 1955, pp, 476-489. 

Strauss, Anselm and Leonard Schatzman. "Cross-class Interviewing." 
An Analysis of Interaction and Communicative Styles." 
Human Organization , 1962, vol. 14 
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Whyte, William F, "Interviewing in Field Research." Human 

Organization Research. Ed, by R,N,A, Adains § J, J, Preiss. 
Homewood, Illinois: Dorsey Press, 1960. 

Whyte, William F, "On Asking Indirect Questions," Human Organization , 
Vol. 12, No, 2, pp, 21-23. 

*Whyte, William F. "Interviewing for Organizational Research," 
Human Organization , summer, 1953, pp- 15-22. 

C. Some techniques and tools for recording qualitative data. 

^Barker, Roger G, and Herbert F. Wright. Midwest and Its Children : 
The Psychological Ecology of an American Town , Evans ton. 111. , 
Row, Peterson § Co., 1954. pp, 195-22-^. 

"Data Recording and Data Retrieval." Issues in Participant Observation , 
Ed. by McCall and Simmons. Reading, Mass., Addison Wesley Pub, 
Co, , 1969, pp. 73-76, 

Murray, Melbin, "Field Methods and Techniques. An Interaction 

Recording Device for Participant Observers." liuman Organization . 
Vol, 13, 1954. pp. 29-33. 

Murray, Melbin. 'flapping Uses and Methods," Humrjn . Organization 
Search, Ed, by Adams § Preiss, Homewood, Illihois; Dorsey 
Press, 1960, 

Schoggen, Phil. 'Mechanical Aids for Making Spocij.'.ent Records of 
Behavior," Child Development , Vol. 25, 1964. pp, 985-988. 

Vaugh, Warren J, Jr. and Emelie Faber, "Field Methods and Techniques, 
The Systematic Observation of Kinderga. ten Cliildren," Himian 
Organization . Vol, 11, 1952, No. 2, summer 1952, pp. 33-36, 

Part IX, Related Methodological Approaches to Qualitative Data 
Collection and Analysis 

A, Survey techniques and qualitative analysis, 

Campbell, Donald T. "Tne Informant in Quantitative Research." 

Issues in Participant Observation . Ed. by McCall and Simmons, 
Reading, Mass., Addison Wesley Pub, Co,, 1969. pp. 289-294. 

*Sieber, Sam D, "The Integration of Field Work and Survey Methods." 
New York, Bureau of Applied Social Research, Columbia University, 
1969, Prepared for a reading in the Sociology of Education. 
The Free Press, (forthcojoing) . 

*Vidich, Arthur J. and Gilbert Shapiro. "A Comparison of Participant 
Observation and Survey Data," Issues in Participant Observation . 
Ed. by McCall and Sijnmons, Reading, Mass., Addison Wesley 
Pub. Co,, 1969, pp* 294-295. 
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B. Quasi- experimental re$earch designs. 



*Cajnpbell, Donald T. and Julian C. Stanley, Experimental and Quasi- 
Experimental Designs for Research . New York, Rand McNally and 
Cot Reprinted from the Handbook o£ Research on Teachdjig, 
1963 by the American Educational Research Association, pp. 34-71. 

C. Ecological and ''Classical" issues in research. 

Gump, Paul V. and Jacob S. Kounin. "Issues Raised by Ecological 
and "Classical" Research Efforts." Merrill -Palmer Quarterly , 
6:145-152, 1960. 

D. Observation and psychological tests. 

*Mensh, Ivan N. and Jules Henry. "Direct Observation and Psychological 
Tests in Anthropological Field Work." I ssues in Participant 
Observation . Ed. by McCall and Smmons, Reading, Mass., 
Addison Wesley Pub. Co.. 1969. pp. 303-321. 

E. Example of an analysis of complex organizations. 

Weiss, Robert S. and Eugene Jacobson. "A Method for the Analysis 

of the Structure of Complex Organizations." American Sociological 
Review , 20:661-668, 1955. 

Part VIII. Final Reporting 

A. Issues related to ethics and politics. 

*Becker, Howard S. "Problems in the Publication of Field Studies." 
Issues in Participant Observation . Ed. by McCall ^and Simmons. 
Reading, Mass., Addison Wesley Pub. Co. 1969. pp. 260-275. 

*Cohen, David K. "Two: Politics and Research: Evaluation o£ Social 
Action Programs in Education." Review of Educational Research . 
40:213-238, April, 1970. 

Kaplan, Abraham. The Conduct of Inquiry. Methodology for Behavioral 
Science. San Francisco, Chandler Pub. Co., 1964. pp. 398-410. 

*McCall .and Simmons, eds. Issues in Part icipant Observation . 

Reading, Mass., Addison Wesley Pub. Co., 1969. pp. 258-260. 

*Rainwater, Lee and David J. Pittman. "Ethical Problems in Studying 
a Politically Sensitive and Deviant Community." Issues in 
Participant Observation . Ed. by McCall and Simmons. Reading, 
Mass., Addison Wesley Pub. Co., 1969. pp. 276-288. 

*Sjoberg, Gideon, ed. Ethics, Politics and Social Research . 
Cambridge, Mass. , Schenlcman, 1967. 
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Appendix G 



Selected Bibliography for A 
Course on Classroom Research 

The references listed below are organized into various categories 
as follows: Theoretical or philophical analyses of concepts involved 
in classroom research, review articles and discussions of methodology, 
non- instrumented studies, some selected research reports and references 
for reports, evaluation studies including classroom observations, and 
sources for different substantive concepts for use in describing 
classrooms. Readings from these categories could accompany the course 
in classroom research described in the body of the report. 

The listings include individual reseai^ch repoi ts only when these 
are not listed in Mirrors For Behavior or not widely known. Many of 
the review articles listed contain excellent biliographies and are 
readily avaialble if the reader desires sources for studies. 



Philosophical or theoretical analyses o£ concepts and te^rms used in 
describing classrooms. 

Aschner, Mary Jane. "The Language of Teaching." B. Othanel Smith and 

Robert H. Ennis. eds. Language and Concept*:, in Education. 
Chicago, Rand McNally, 1961 p, 112-126. 

Getzels, Jacob W. and Herbert A. Thelem 'The Classroom Group as a 
Unique Social System." National Society for the Study of 
Education . The Dynamics of Instructional Groups, Fifty-nintli 
Yearbook, Part II . Chicago, University of Chicago Press, 

1960. p. 53-82. 

Scheffler, Israel, "Knowledge and Teaching." Conditions of Knowledge . 
Fairlawn, N. J. , Scott, Foresman, and Co., 1965 p. 7-22, 

Smith, B. Othanel. "A Concept of Teaching." B. Othanel Smith and 
Robert H. Ennis, eds. Language and Concepts in Education . 
Chicago, Rand McNally, 1961. p. 86-101* 

Swift, Leonard, "rxplanation." B. Othanel Smith and Robert H. Ennis, 
eds. Language and Concepts in Education. Chicago, Rand McNally, 

1961. p. 179-194. 

Travers, Robert M. "Towards Taking the Fun Out of Building a Theory 
of Instruction." Teachers College Record , 68:49-60, October, 
1966. 

Ifeick, Karl E. "Systematic Observational Methods." Gardner Lindzey 
and Elliot Arorsun, eds. Handbook of Social Psycholog y. 
2nd ed. Reading, Mass., Addison-Wesley , Vol. II, 1968, 
p- 357-451. 
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The Following references cojuprise an anthology containing the 
research reports, including analyses systejns and coding instructions, 
of approximately 80 classroom observation systems. 



Simon, Anita and E. Gil Boyer, eds. Mirrors For Behavior > New York, 
Research for Better Schools, Vols. I-VI, 1967; Vols. VII-XIV 1970. 

Simon, Anita and E. Gil Beyer, eds. Mirrors For Behavior, Suinmary . 
New York, Research for Better Schools,' 1970. 

Articles or volumes which emphasize the theoretical and methodological 
issues in or relevant to classroom observation and research strategies 
using selected research reports for exan^iles. Includes reviews with 
good bibliographies. 

Baldwin, Clara P. "Naturalistic Studies of Classroom Learning." 
Review of Educational Research , 35:107-113, 1955. 

Bel lack, Amo A. "Methods for Observing Classroom Behavior of Teachers 
and Students." Paper presented at the conference on "Methods 
of Determining Criteria for the Evaluation of Comprehensive School 
Sponsored by Pedagogishes Zentrum Berlin, November 12-15, 1968. 

Biddle, Bruce J. 'T^ethod and Concepts in Classroom Research." 
Review of Educational Research , 37:337-357, 1967. 

Biddle, Bruce J. and Raymond S. Adams. 'Teacher Behavior in the 
Classroom Context." Lawrence Siegel, ed. Instruction : 
Some Contemporary Viewpoints . San Francisco, Chandler, 1967. 
p. 99-136. 

Biddle, Bruce J. and Elena J. William, eds. Contemporary Research on 

Teacher Effectiveness . New York, Holt, Rinehart and Winston, 1964 

Gage, Nathaniel L. , ed. Handbook of Research on Teaching . Chicago 
Rand McNally and Co., 1963* 

Gage, M. L. and W. R. Unruh. "Theoretical Formulations for Research 
on Teaching." Review of Educational Research , 37:358-370, 1967. 

Greenberg, Selma B. "A Comparative Analysis of Selected Studies, of 
Classroom Teaching." Ed. D. Dissertation. New York, TeacEiers 
College, Columbia University, 1966. Typewritten. 

Hoetker, William J. and Wil.Tiam P. Anlbrand, Jr. "The Persistence of 
the Recitation^" American Educatio nal Research Journal, 
6:145-167, 1969. " 

Meux, Milton 0. "Studies of Learning in the School Setting." 
Review of Educational Research, 37:53^-562, 1967. 
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Nuthall, G. A. "A Review of Some Selected Recent Studies of Classroom 
Interaction and Teachijig Behavior/^ Robert Stake, ed. Classroom 
Observation . American Educational Research Association 
Monograph Series on Curriculum Evaluations No. 6, Chicago, Rand 
McNally, 1970. p. 6-29, 

Rosenshine, Barak. '"Evaluation of Classroom Instruction.'' Review of 
Educational Research , 40:279-300, 1970. 

Rosenshine, Barak. "'Some Criteria for Evaluating Category Systems; 
An Application to the Topic Classification System." Robert 
Stake, ed. Classroom Observation. American Educational Research 
Monograph Series on Curriculum Evaluation, No. 6, Qiicago, kanJ 
McNally, 1970. p. 109-124. 

Non- instrumented studies of classrooms. 

Henry, Jules. "Attitude Organization in Elementary School Classrooms." 
American Journal of Orthopsychiatry , 27:117-133, 1957. 

Henry, Jules. "Docility, Or Giving Teacher What She Wants." Journal 
of Social Issues , 11:33-41, 1955. 

Jackson, Philip W. Life in Classrooms . New York, HOlt, Rinehart 
and Winston, 1968. 

Jackson, Philip W. "The Way Teaching Is." Monograph of the Association 
for Supervision and Curriculum Development of the National 
Education Association, 1966. p. 7-27. 

Smith, Louis M. and William Goeffrey. The Complexities of an Urban 
Classroom. New York, Holt, Rinehart and Winston, 1968. 

Selected research reports not reproduced in Mirrors for Behavior 



Pey, James T. Patterns of Verbal Communication in Mathmatics Classes, 
New York, Teachers College Press, 1970. 

Cordis, Felice W. "A Piagetiaa Analysis of the Teaching of Seriation. 
Concepts in Fcur Grade Classrooms" Ph.D. Dissertation. New 
York, Teache.\^s College, Columbia University, 1970. Typewritten. 

Lambert, Nadene M. , Hugh W. Cox and Carolyri Hartsough. "The Observa- 
bility of Intellectual Functioning of First Graders." 
Psychology in the Schools , 7:74-85, 1970. 

Gump, Paul J. Classroom Behavior Setting: I-s Nature and Relation to 
Student Behavior , U.S. Ot'fice of Education, Final Report ^ 
Project No. 2453, 1967. Microxiche No. Ed 015 515 

Studies of educational settings which include classroom observations* 
among the data collected. 
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Almy, Millie and Associates. Logical Thiaking in Second Grade > 
New Y"^lv, Teachers College Press, 1970^ 

Chall, Jeanne S. and Shirley C. Feldman. A Study of Depth, of First- 
Grade Readings An Analysi s of the Interactions of Professed 
Methods. Teacher Iinplementation, and Child Background s City 
College of the City of New York, 1966. Cooperative Research 
Project No. 2728. Microfiche No. Ed. 010-036. 

Child Development Research and Evaluation Center for Head Start. Temple 
University, Annc^al Report , Contract No. OE 04126, p. 138-167. 
Microfiche No. Ed. 030 487. 

Harris, Albert J. and B.L. Serwer. Comparison of Reading Approaches in 
First Grade Teaching with Disadvantage Children . (The CRAFT 
Project) . Cooperative Research Project No. 2677, New York, 
City University of New York, 1966. Microfiche No, Ed QIC 037. 

Hunter, Elizabeth. "Talking in First Grade Classroom." Urban Review , 
4:39-43, October, 1969. 

Minuchin, Patricia, Barbara Biber, Edna Shapiro and H. Zimiles. The 
Psychological Impact of School Experienc e. New York, Basic 
Books, 1969. 

Neujhar, James L. "An Analysis of Teacher-Pupil Interactions When 
Instruction is Individualized." Ed,D Dissertation. New York, 
Teachers College, Columbia University, 1970. Type;inritten. 

Prescott, Elizabeth. Group Day Care As a Child-Rearing Environment . 
An Observational Study of Day Care Programs , Pasadena, Calif. , 
Pacific Oaks College. Microfiche No. Ed 024 453 and Ed 021 363. 

Scanlon, Robert G. Factors Associated with a Program for Encouraging 
Self -Initiated Activities by Fifth and Sixth Grade Students in a 
Selected Elementary School Emphasizing Individualized Instruction . 
Pittsburgh University School of Education. Microfiche No. Ed 
015 785. 

Examples of sources for ne\f substantive concepts to use in describing 
classrooms. 

Bruner, Jerome S. , Rose Olver and Patricia Greenfield. Studies in 
Cognitive Growth . New York, John Wiley, 1966. 

Cazden, Courtney. ''Evaluation of Learning in Preschool Education: Early 
Language Development." Benjamin S. Bloom, J. T3i.omas Hastings, 
and George F. Madaus, eds. Handbook on Formative and Sumniative 
Evaluation of Student Learning . New York, McGraw Hill, 1971. 
p. 347-398. ~" 

Gall, Meredith D. "The Use of Questions in Teaching." Review of 
Educational Research , 4:707-719, 1970. 
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Labov, William* The Stnady of Non- Standard English , National Council 
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APPENDIX H 



This course has been designed on several levels. The overall aijn 
o£ the session is to prepare individmls for v/ork in research divisions 
in boards of education. By' A'lrtue of previous course worlc and experi- 
ence, you have developed, a mornber of skills relevant to educational 
research. It is hoped that in this course you will bring these skills 
and others to bear in order to produce a finished research product. 

The organization of exposure is under two headings: One experi- 
mental design, will be handled by Mr, Diamond and Mr, Pyrczak, The other 
section, data jnanagement will be disucssed by Mr, Ashler and Miss 
Byers, (These are the general emphases only. Each group of instructors 
will, of course, have specific areas of concern,) 

The section on experimental desigr will be built around indivi- 
dval research projects. Each student v/ill be required to write a 
proposal for a given prc'ject. In addition, a final report will be 
written following tlie analysis of ciata. The proposal and the final 
report will represent "^he output from the experimental design section 
of Education 864, Specific details o£ these reports and the course 
sequence will be given during class meetings. 

Although Mr, Ashler will be working with the class each day, Mr, 
Diamond and Mr. Pyrczak will meeting, with the entire class, approxi-. 
mately twice weekly, on the average. At these meetings, special topics 
of interest will be discussed and students will share their thoughts 
with other members of the group. 
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APPENDIX I 
OUTLINE OF COURSE 864-11 
SUMMER SCHDOL, 1970 - SECOND SESSION 
University of Pennsylvania 

I. Introduction to coinputer operations and general prograinming 
techniques, using the Datel low- speed terminal 

A. Visit to Coinputer Center and Dietrich Hall remote terminal 

B* Direct interaction with the computer, using Conversational 
Programming System (CPS) programs (e.g. like the one in 
which an individual plays tic-tac-toe with the computer) 

C. Writing of small programs in CPS language (similar to 
FORTRAN IV) 

1. Get the sum of ten nur.ibers 

2. Generate a Fibonacci series of numbers 

3. Compute the average of a series of positive numbers 

4. Determine the maximum and minimum of that series 

5. Determine the number nearest the mean A^lue of that 
series 

II. Writing and running programs 'vritten in FORTRAN IV and running 
statistical (canned) programs 

A- Characteristics of FORTRAN IV conpared with those of CPS 

1. Additional features of FORTRAN TV (e.g. FORMAT state- 
ments, array manipulation) 

B. Illusn^ations of useful techniques 

1. Selecting and combining data, read from two cards per 
case, to produce **new** information on a single card per 
case; and simultaneously printing counts and totals 

C. Use of statistical prograins, especially EMD's 

1. Preparation of data prior to conputer processing 

2. Discussions of debugging methods 

3* 'TDeciphering** typical program descriptions 
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D. Basic Job Control Laxtguage (JCL) needed to run typical pro- 
grams ^control cards for the operating sy^s tejn 

1. Description of function of each t)T3e of system control 
card 

2. Description of peripheral (disk and tape) data storage 
a. Applications of data retrieval techniques using JCL 

3. Running of inulti- itep jobs 

E. Preparation and submitting of FORTRAN IV jobs using both 
the low- speed terminal and the Cpunched) card reader 

Miscellaneous topics covered 

A. Optical Scanning techniques for producing punched data 
cards from handmarked sheets; e.g. test answer sheets 

B. Data reduction techniques for preparing large batches of 
data for computer processing. 

C. Utility programs and their many uses in data handling 
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.APPENDIX J 

Tentative Schedule* 
Education 864 SSI I 

University of Pennsylvania 

I . Introduction 

II. Assignment of projects 

A. Preparation for role playing inter^^iew 

B. Proposal outline 

III. Planning a research study 

IV. Experimental Design and Analysis 

V. Overview of Measurement Theory 
Vr. Special Topics 

A. ANOVA 

B. iMeasurenient of Change 

C. Regression 

D. Unit of Analysis 
VII. Final report outline 



*The order given here is not necessarily the order in v^^iich the course 
will be given. In addition, as each person eowks with Mr. Pyrczak 
or Mr. DiaiDond special coverage, suitable to a given project, v/ill be 
considered. 
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APPENDIX K 



ujvr^ERSirr of Pennsylvania 

PHILADELPHIA 19104 



August 7, 1970 

Wharton School of 
Fifiance /wd Cofnmerce 

Wharton Compi'tational 
Services 



Mr, Irvin Farber 
Room 400 

School District of Philadelphia 
21st and Parkway 
Philadelphia, Pn, 19103 

Dear Irv; 

As you suggested I am setting forth in this letter some thoughts about 
the computer course concluded today which was sponsored by the School 
District for the benefit of a group of trainees. 

You already have an outline of the course as presented ♦ As can be seen 
from that outline, the topics included both programming topics and the 
use of canned programs and operating system facilities. None of the 
trainees had prior programming experience; the introduction to program- 
ming was , therefore, on a rather elementary level; however, I believe 
that such important concepts as memory organization, looping and iterative 
processes, tolerances, etc, were assimilated adequately. 

These train<2es could now reasonably be expected to write short programs 
to transform raw data into a form suitable for processing by canned 
programs. For example, they should be able to arrange. for the merging 
of data on two independent decks of cards. They should also be able to 
program straightforward computations. 

In addition to programming concepts, the trainees were given practice 
in the use of programs from the Bio^-Medical series and the University 
of Miami library. Data was submitted to these programs both in card 
form and in the fom of files "resident on the magnetic disk peripheral 
memory of the Univers:Lty comDUting system. The Job Control Language 
necessary to store and access data on the magnetic disks was discussed, 
and one may reasonably expect that the trainees could now, with 
minimal help from a professional programmer, set up computer jobs 
involving these functions. 

Not every aspect of the course was equally successful. It is my 
feeling that the use of individual interactive terminals early in the 
course was not justified in terms of the insights obtained thereby, 
A more structured course organization, particularly during the 
first week, would probably have been better, inasmuch as there was some 
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Mr . Irvin Farber 
Auf^ust 10, 1970 
2 

'is. 

tendency on the oart of t'?.e trainees to flounder and not make as effective 
use of the time P-^rc facilities as va? expected , des oi te the presence of t\-:c 
very competent course .5'- pi -^.t ants to -provide abundant individual help and 
counselling. 

In my opinion^ any future course of this sort should have a clearly desip,- 
nated daily time allot-nent, either 9:00 A.M. to :^oon or 1:00 r.M. to 
41:00 P.M., without intcrruotion . 

In view of the time reruirec for most oeople to master elcniGntarv nrof-^ram-- 
ming concepts, and in view of the degree to which such mastcr^.^ facilitates 
the use of system facilities and canned programs ^ even where no actual 
programming Is-: contemplated, it is my belief that persons entering a course 
in the applications of computers to the problems of educational research 
should have a prior programme -ig background. It would be most desirable 
that they take a course in a programming language, e.g. Fortran, prior to 
this course. In future years it might be worth considering the presentation 
of a course equivalent to Education 865, Mass Data Processing, during the 
first summer session, and the j- resentation of a course specifically oriented 
to the use of computers in educational research during the second summer 
session. Of course, persons already familiar with a programming language 
could omit the earlier course. 

T intend to prepare some additional materials to be mailed to the trainees. 
As soon as these are ready, I will send you a copy. 

Sincerely yours, 

^■•Daniel Ashler 
Director of Computational Services 

DA/a : 
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APPENDIX L 



UJUVERSITY of FENKSYLVAKIA 

PHILADELPHIA 19104 



Graduate School of Educaticn 
3700 Walnut Strtet 

Educational i<zsEARCH and S,»vRvick Tup^au 

August li, 1970 

Dr, Irv Farber 
Division of Research 
Board of Education 
21st The Parkway 
Philadelphia, Penna. 

Dear Irv, 

Let me put down some thoughts concerning the summer workshop, as you 
requested. 

First, in terms of what the students should be able to do and what was 
theoretically learned: 

1. They should have ^ basic understanding of statistical inference. 

By this, I mean, they can define and discuss concepts such as samp- 
ling distribution, null and alternative hypotheses. Type I, Type II 
errors, power, standard error, and so on. 

2. They can discuss different types of research, experimental and natu- 
ralistic, 

3. They have been exposed to the philosophy and rationale underlying 
analysis of variance, one and two way interactions, ordinal and dis- 
ordinal; analysis of covariance, Sc.iie people worked on the multi- 
variate analysis of variance on their own. If the course fell down 
anyplace, it was in the analysis section. This was partly due to my 
misperception of what these people had been exposed to in the way of 
background information in statistics. 

4. Within the constraint specified in #3, I believe that these people 
could write a proposal for any research project in which the School 
District would engage. 

5* Other areas discussed: robustness, regression toward the mean, the 
unit of analysis, and the matching fallacy. 



ERIC 



-137- 



yty own feeling is that none of them would be able to function as an Associate, 
but would be a real asset as an Assistant to somebody of, say, 
caliber. Maybe my standards are too high. 

I would m;ake the following recommendations in the event the workshop is attempted 
again, 

1. Have more examples of actual research and how people handle "pitfalls," 

2. Have more communication among students concerning their projects so 
they can learn from each other a 

3. Coordinate efforts between sections to a greater degree. 

4. Have a review of basic statistical concepts at the start of the course. 

In retrospect, I see that, although we had some of this, there was not enough. 
I hope this was the sort of thing you had in mind. 



Cordially , 




James Diamond 
Assistant Professor 



JD/bh 
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